Are Intermediate Layers and Labels Really Necessary? A General Language Model Distillation Method
Year: 2023

Abstract:
The large scale of pre-trained language models poses a challenge for their deployment on various devices, with a growing emphasis on methods to compress these models, particularly knowledge distillation. However, current knowledge distillation methods rely on the model’s intermediate layer features and the golden labels (also called hard labels), which usually require aligned model architecture and enough labeled data respectively. Moreover, the parameters of vocabulary are usually neglected in existing methods. To address these problems, we propose a general language model distillation (GLMD) method that performs two-stage word prediction distillation and vocabulary compression, which is simple and surprisingly shows extremely strong performance. Specifically, GLMD supports more general application scenarios by eliminating the constraints of dimension and structure between models and the need for labeled datasets through the absence of intermediate layers and golden labels. Meanwhile, based on the long-tailed distribution of word frequencies in the data, GLMD designs a strategy of vocabulary compression through decreasing vocabulary size instead of dimensionality. Experimental results show that our method outperforms 25 state-of-the-art methods on the SuperGLUE benchmark, achieving an average score that surpasses the best method by 3%. 1

heading: 1 Introduction
text: The exponential increase in the scale of pre-trained language models has impeded their deployment on a wider range of devices. To mitigate the inference cost of large-scale pre-trained language models, researchers have increasingly focused on model compression methods, aiming to compress a large
*This work was done when the author visited Zhipu.AI. †Corresponding authors. 1The code is available at https://github.com/aitsc/
GLMKD.
model into a small one with as little performance loss as possible (Li et al., 2022). While model compression can yield very small models, maintaining performance without degradation is still a challenging task, particularly when the large and small models have a significant discrepancy in parameter size (Li et al., 2021). There are various methods of model compression (Wang and Yoon, 2022), including network pruning (Huang et al., 2022), quantization (Boo et al., 2021), neural architecture search (Elsken et al., 2019), parameter sharing (Lan et al., 2020), matrix decomposition (Tahaei et al., 2022), and knowledge distillation (Liu et al., 2022b; Zhang et al., 2022) etc. Currently, knowledge distillation is an important research direction, which allows for the transfer of knowledge from a large model (the teacher) to a small one (the student).
There are two main optimization objectives of the earliest knowledge distillation methods (Hinton et al., 2015): increasing the similarity between the student’s prediction probabilities for the task and those of the teacher (soft targets); increasing the similarity between the student’s predictions and the golden labels (hard targets). When the knowledge distillation method is applied to language models, there are typically two directions for improvement: leveraging the intermediate layer features of the teacher model, such as hidden states and attention, to obtain additional hidden state knowledge (Sun et al., 2019; Liu et al., 2022a); and refining the two objectives (soft targets and hard targets) and weights of objectives (Lu et al., 2021; Zhou et al., 2022). As shown in Table 1, these methods all rely on intermediate layer features or hard labels of the model. However, using intermediate layer features and hard labels is often accompanied by certain limitations, such as the requirement for the teacher and student models to have the same structure and dimensions, or the need for additional data and labels. These limitations make the implementation
9678
of distillation complex and hinder the applicability of these methods to a wider range of models and data. Moreover, existing methods often reduce the parameter scale of the model by decreasing the number of layers and hidden dimensions, neglecting the impact of vocabulary size.
To address these problems, we propose a general language model distillation (GLMD) method that performs two-stage (pre-training and task-specific stages) word prediction distillation and vocabulary compression. Specifically, GLMD distills the model using only the language modeling word prediction logits during the pre-training stage, which is similar to the soft labels used in general methods. The key to this stage is that we distill both masked and unmasked tokens. In the task-specific stage (fine-tuning), GLMD distills both the language modeling word prediction logits and the soft labels. The language modeling word prediction logits is crucial in this stage, making the distillation more consistent between the pre-training and task-specific stages. In these two stages, GLMD eliminates the need for complicated intermediate layers and golden labels and does not require the
selection of intermediate layers or labeled dataset. Meanwhile, GLMD uses the teacher vocabulary to map low-frequency words to the most similar highfrequency words, further compressing the model with almost no performance loss.
In summary, our major contributions are:
• We propose a general language model distillation (GLMD) method that saves the tedious work on intermediate layer features and golden labels, and does not require the selection of intermediate layers or labeled dataset. We demonstrate through analysis that GLMD allows models to autonomously learn intermediate layer features that are similar to those of the teacher.
• We propose a vocabulary compression strategy based on the long-tailed distribution of words in data, which reduces the vocabulary size without reducing dimensions of the model. Additionally, our vocabulary compression strategy can be used in conjunction with other dimensionality reduction strategies with very little performance loss.
• We verify that GLMD outperforms 25 stateof-the-art model distillation methods on the SuperGLUE benchmark, achieving an average score that surpasses the best method by 3%. Furthermore, our vocabulary compression strategy also outperforms other 2 dimensionality reduction strategies. We also investigate distillation of ultra-large-scale language models (10B-scale) for the first time.

heading: 2 Related work
text: Language Model Distillation Since the introduction of knowledge distillation to pre-trained language models by PKD (Sun et al., 2019), an increasing number of researchers have recognized the importance of knowledge distillation. During the early stage of the research, PD (Turc et al., 2019) employed simple baseline (soft targets) distillation for language models, resulting in a relatively limited transfer of knowledge for the model. Subsequent research had primarily focused on the use of intermediate layer features in language models (Xu et al., 2020; Sanh et al., 2019), including distillation of models during pre-training stage (Sun et al., 2020b), task-specific stage (Aguilar et al., 2020), and two-stage (Jiao et al., 2020) approaches. Given
the typically large amount of intermediate layer features, some work had utilized features from only a single intermediate layer (Wang et al., 2020, 2021), while other work had examined methods for reducing the scale of features (Liu et al., 2021). Recent work has explored ways to utilize better intermediate layer features, for example, CoDIR (Sun et al., 2020a) and LRC-BERT (Fu et al., 2021) utilized cross-sample feature relationships through contrastive learning; ALP-KD (Passban et al., 2021) and Universal-KD (Wu et al., 2021b) combined all intermediate layer features through attention mechanisms; Meta-KD (Pan et al., 2021) and HRKD (Dong et al., 2021) used meta-learning to assign appropriate weights to intermediate layer features; RAIL-KD (Haidar et al., 2022) randomly selected different intermediate layers for distillation; CKD (Park et al., 2021) and MGSKD (Liu et al., 2022a) used some variety of similarity calculation methods for intermediate layer features; DIITO (Wu et al., 2022) allowed student models to learn counterfactual outputs by swapping intermediate layer features between different samples.
However, the use of intermediate layer features has additional limitations, such as requiring the same model structure (Sun et al., 2020b) for both teacher and student, or requiring linear transformations (Jiao et al., 2020) to ensure consistency in dimensions between teacher and student. There were also methods that only used soft and hard targets, for example, Annealing-KD (Jafari et al., 2021) and Continuation-KD (Jafari et al., 2022) gradually increased the weight of soft targets through simulated annealing; RW-KD (Lu et al., 2021) adjusted the weight of soft and hard targets through meta-learning and a dev set; MetaDistil (Zhou et al., 2022) allowed the teacher to learn how to output better soft labels through meta-learning and a quiz set. These approaches relied on hard labels and may have even required additional datasets for partitioning. Additionally, there had been approaches that distilled multiple teachers (Yang et al., 2020; Wu et al., 2021a; Yuan et al., 2021; Li et al., 2021) or teacher assistants (Mirzadeh et al., 2020; Son et al., 2021) at the same time, but they still relied on intermediate layer features or hard labels. In comparison, GLMD can achieve the strongest performance in a more broadly applicable context without intermediate layer features or hard labels.
Vocabulary Compression Vocabulary compression refers to reducing the parameter size of the
vocabulary in a language model. In the knowledge distillation of language models, reducing the parameter size of the model is mainly used to reduce the number of model layers or dimensions (Jiao et al., 2020; Wang et al., 2021). MobileBERT (Sun et al., 2020b) and ALBERT (Lan et al., 2020) independently reduced the dimensions of the vocabulary to achieve vocabulary compression. MobileBERT needed to restore the dimension of the vocabulary in the calculation of the pre-training loss due to the requirement to ensure consistency between the vocabulary dimension and the model output dimension. On the other hand, ALBERT used a linear layer to alter the output dimension of the model. However, these vocabulary compression methods only reduced the dimensionality and ultimately required dimensionality restoration. In contrast, our vocabulary compression method reduces the number of words through mapping, further compressing the model with almost no impact on performance.

heading: 3 Preliminaries
text: In this section, we introduce the objective function for knowledge distillation of language models, and formalize the language modeling word prediction logits.

heading: 3.1 Knowledge Distillation
text: Knowledge distillation aims to transfer the knowledge of the teacher T to the student S. The knowledge and transfer method can be formalized as model features and distance metrics respectively. Formally, knowledge distillation for language models typically consists of the following three objective functions:
Lsoft = τ2KL(σ(fSl (x)/τ), σ(fTl (x)/τ)) Lhard = CE(σ(fSl (x)),y) Linter = d(fS(HS), fT (HT )) (1)
where τ denotes the softening parameter (temperature), KL(·, ·) denotes the KL divergence, σ denotes the softmax function, x ∈ Rl denotes the input sequence (token ids) of length l for the language model, fSl (x) and f T l (x) denote the logits output by the student and the teacher before computing the task loss respectively, CE(·, ·) denotes the cross entropy, y denotes the hard labels, d(·) denotes the distance metric (e.g., KL divergence and mean square error), HS and HT denote the intermediate layer features (e.g., hidden states and
attention) of the student and the teacher respectively, fS(·) and fT (·) denote custom transformations (e.g. linear transformations) of the student and teacher features, respectively.
Currently, mainstream methods employ different combinations and weighting schemes of the three objective functions in the pre-training and taskspecific stages. For example, TinyBERT (Jiao et al., 2020) optimizes Linter in the pre-training stage and optimizes Linter and Lsoft in the task-specific stage, while MetaDistil (Zhou et al., 2022) only optimizes Lsoft and Lhard in the task-specific stage. Notably, to ensure feature dimension matching between the teacher and student, Linter relies on complex custom transformations, such as linear transformations (f(H) = WH) and pair-wise scaled dot-product (f(H) = HHT / √ dimensionality). In contrast, our method does not rely on Linter and Lhard.

heading: 3.2 Language Modeling Word Prediction Logits
text: Language modeling typically refers to unsupervised tasks in the pre-training stage, such as causal language modeling for GPT (Radford et al., 2018), masked language modeling for BERT (Devlin et al., 2019), and autoregressive blank filling for GLM (Du et al., 2022). This process typically requires a decoder to decode the model’s output into a prediction logits for each word. The decoder is typically a linear transformation using the vocabulary parameters as weights. The language modeling word prediction logits can be formulated as follows:
LM(x) = ft(x)W T v (2)
where Wv ∈ Rv×h denotes the vocabulary parameters (weight of embeddings-layer), and ft(x) ∈ Rl×h denotes the output of the final layer of transformer. The scalar value l denotes the length of the text sequence, v denotes the number of tokens in the vocabulary, and h denotes the dimensionality of the hidden layer. It is worth noting that the LM(·) can also be computed at the task-specific stage.

heading: 4 Method
text: In this section, we propose a general language model distillation (GLMD) method with two-stage word prediction distillation and vocabulary compression. Figure 1 shows the overview framework of GLMD, which implements a vocabulary compression strategy while performing a two-stage
word prediction distillation process. We next provide a detailed description of these two components.

heading: 4.1 Two-stage Word Prediction Distillation
text: To eliminate the reliance on intermediate layer features and hard labels in distillation, we propose a two-stage word prediction distillation process based on the language modeling word prediction logits. It allows teacher models and students models to have different model structures and does not need the selection of intermediate layers. This process makes the distillation goal more closely aligned with the model’s task and makes the model’s distillation more consistent across both the pre-training and task-specific stages. During the pre-training stage, we optimize the student model with the objective function L′sp. We then optimize the student again using L′sp during the task-specific stage. After these two training phases, we finally optimize the student with the objective function Lst from the task-specific stage. Our objective functions Lst and L′sp are defined as:
Lst = Lsoft L′sp = Lsp ⊙mp Lsp = τ2KL(σ(LM S(x) τ ), σ( LMT (x) τ ))
(3)
where ⊙ denotes the Hadamard product, and mp ∈ Rl denotes the mask vector, which only masks the pad while preserving the masked and unmasked tokens. We find that unmasked tokens are typically not predicted by language modeling, but they can provide more knowledge for distillation. L′sp and Lst represent the soft targets for the pre-training and task-specific stages, respectively. It is worth noting that L′sp can be used in both pre-training and task-specific stages, making the optimization objectives more consistent across the two stages.

heading: 4.2 Vocabulary Compression
text: To further compress the parameter scale of the model, we propose a vocabulary compression strategy that reduces the number of tokens in the vocabulary. Because word frequencies have a longtailed distribution, some low-frequency words can still be understood by the language model after being replaced with similar words. Let the compression rate of the vocabulary be rv, and the number of all tokens before compression be v. We sort the tokens in the pre-trained corpus according to their frequency of occurrence. The tokens ranking in the top vrv are treated as the compressed tokens wc ∈ Rvrv , while the remaining tokens wm ∈ Rv(1−rv) are to be mapped. Figure 1 illustrates the key aspect of our vocabulary compression strategy, which includes replacing low-frequency words with similar words through token mapping and aligning the weight matrix of the embeddinglayer through weight mapping. The token mapping aims to map wm ∈ wm to wc ∈ wc, and the mapping function is defined as:
ftm(wm) = argmax wc∈wc (Sim(v(wm), v(wc))) (4)
where v(wm) and v(wc) denote the token vectors of wm and wc in the pre-trained teacher’s vocabulary weight Wvt , respectively. Sim(·, ·) is a function of calculating similarity using the inner product, which is similar to the decoding in Equation 2. The weight mapping aims to remove wm from Wvt , and the mapping function is defined as:
fwm(Wvt) = Wvt [wc] (5)
where [·] denotes a slicing operation, specifically obtaining all vector of wc ∈ wc from Wvt .

heading: 5 Experiments
text: In this section, we demonstrate the effectiveness of the GLMD on models with different parameter scales (110M, 340M, and 10B) and analyze the role of different components and why they are effective. All experiments were conducted on 40 NVIDIA A100 GPUs and completed within 4 months, utilizing the PyTorch framework.

heading: 5.1 Experimental Setup
text: Datasets To evaluate our GLMD method, we conduct experiments on the more challenging SuperGLUE (Wang et al., 2019a) benchmark instead of GLUE (Wang et al., 2019b). The use of more difficult tasks allows for a better display of the discrepancy between different distillation methods. We use the average score across 8 tasks in SuperGLUE as the evaluation metric. We use BooksCorpus (Zhu et al., 2015) and English Wikipedia as the data (19GB) for the distillation of pre-training stage for all methods.
Baselines We compare 25 commonly used distillation methods as listed in Table 3. We provide a more detailed description of these methods in Appendix A.1.
Language Models Student and teacher models of all methods have the standard GLM (Du et al., 2022) architecture. GLM (General Language Model) is a more advanced language model that inherits the advantages of both autoencoding and autoregression. We choose GLM for two reasons: it performs stronger than the commonly used BERT (Devlin et al., 2019) and RoBERTa (Liu et al., 2019); it has open-source pre-trained language models with 10B-scale or even 100B-scale
parameters (Zeng et al., 2022). All pre-trained models, with the exception of MobileBERT, which was trained by us based on GLM, were obtained from the official GLM website 2. Both the teacher and student models were trained with half-precision floating-point (fp16). The model sizes used in this paper are shown in Table 2.
Hyperparameters For our method, the temperatures τ for the loss L′sp and Lst are set to 15 and 1, respectively. All baselines use the best parameters from their respective papers. For all methods that rely on the pre-training stage, the batch size, peak learning rate, and number of iterations are set to 64, 4e-4, and 150000, respectively. For all singleteacher methods, we use grid search to find the best parameters during the task-specific stage, including the learning rate {5e-6,1e-5,2e-5} and batch size {16,32}. The multi-teacher and teacher assistant methods are similar to the single-teacher methods in the core method, with differences in the weighting and assistant of teachers. The other parameters for the task-specific stage are kept consistent with the fine-tuned teacher, using the best parameters provided by GLM. The results for all experiments (w/o T3-S3) are the average of 3 random seeds. For more details on the hyperparameters, refer to Appendix A.2.

heading: 5.2 Main Results
text: In Table 3, we report the average scores of all methods on the SuperGLUE dev set. GLMD−vc denotes GLMD without vocabulary compression strategy. GLMD−vc+mo and GLMD−vc+al denote the use of MobileBERT and ALBERT vocabulary compression strategies on GLMD, respectively. GLMD+al denotes the combination of ALBERT and our vocabulary compression strategies on GLMD.
GLMD achieves the highest performance among 25 baselines on T1-S1, T1-S2, and T2-S2 scales, with a 0.1%, 0.1%, and 3.1% improvement over the best method (TinyBERT), respectively. More importantly, in a fair environment without vocabulary compression, GLMD−vc outperforms the best method by 0.7%, 0.7%, and 3.0%, respectively. This demonstrates that high-performance distillation does not necessarily require intermediate layer features or hard labels, whether reducing the number of layers or the dimensionality of the student model. GLMD significantly outperforms TinyBERT in the distillation process on the scale of
2https://github.com/THUDM/GLM
10B to 2B, indicating that TinyBERT is not suitable for ultra-large-scale model distillation on the SuperGLUE benchmark. The use of vocabulary compression in GLMD still maintains strong competitiveness in further compressing the model. GLMD outperforms the best vocabulary compression strategy (GLMD−vc+al) by 0.1% on T1-S1 scale, confirming that reducing the vocabulary size is an effective strategy. It is worth noting that our vocabulary compression strategy can be combined with other dimensionality reduction methods, such as GLMD+al, which can maintain the original performance even with only one-fourth of the vocabulary parameters. Additionally, some recent baselines did not show the strongest performance, we discuss more factors affecting baseline performance in appendix B.

heading: 5.3 Ablation Study
text: After having validated the effectiveness of GLMD and GLMD−vc, we further analyze in Table 4 the key design factors that impact the performance of the two components in greater detail. (1) Twostage word prediction distillation. The results indicate that both removing mp (row 4) or removing unmasked tokens (row 5) from mp do not perform as well as GLMD−vc (row 3), which confirms the effectiveness of mp in L′sp. The use of mp in L′sp in the task-specific stage makes the distillation of the student more consistent in the pre-training and task-specific stages, which is verified by row 6. The performance degradation observed upon incorporating intermediate layer features (row 7) or hard labels (row 8) into the loss function in GLMD−vc further confirms that such features and labels are not necessary. Additionally, we find that the KL divergence performed better than the MSE (mean square error) in both L′sp and Lst (rows 9 and 10). (2) Vocabulary compression. In addition to mapping low-frequency tokens to similar tokens using the decoder approach, we also attempt to use Cosine similarity (row 13), Euclidean distance (row 14), and direct replacement with [UNK] (row 15) to map similar tokens. We found that these mapping methods did not perform as well as GLMD (row 12), which may be because the mapping method used in GLMD is closer to the decoding approach used in language modeling task. The result of line 12 outperforming line 16 verifies that token mapping is only applicable for students.
=1 =5 =10 =15 =20 MSE KL
Att1 Att2 Att3 Att4 Att5 Att6 Emb HS1 HS2 HS3 HS4 HS5 HS6
K1 K2 K3 K4 K5 K6 sp Q1 Q2 Q3 Q4 Q5 Q6 V1 V2 V3 V4 V5 V6
-0.93 0.82 0.93 0.93 0.92 0.91 -0.21 -0.39 0.02 -0.17 -0.38 -0.33 -0.14 -0.78 -0.46 -0.28 -0.27 -0.27 0.07 0.84 -0.95 -0.76 -0.85 -0.82 -0.71 -0.54 0.5 0.87 0.85 0.72 0.64 0.59 0.03 0.88 0.89 0.89 0.93 0.93 -0.63 -0.22 -0.18 -0.31 -0.44 -0.76 -0.66 -0.8 -0.42 0.37 -0.06 -0.17 0.09 0.89 -0.94 -0.75 -0.8 -0.75 -0.69 -0.28 0.51 0.88 0.86 0.72 0.65 0.59 0.38 0.86 0.87 0.87 0.93 0.93 -0.92 -0.21 -0.19 -0.32 -0.45 -0.77 -0.8 -0.8 -0.41 0.39 -0.06 -0.17 0.09 0.91 -0.94 -0.74 -0.8 -0.75 -0.7 -0.26 0.51 0.88 0.86 0.72 0.65 0.59 0.49 0.86 0.86 0.87 0.93 0.93 -0.92 -0.21 -0.2 -0.32 -0.46 -0.78 -0.81 -0.8 -0.41 0.4 -0.06 -0.17 0.09 0.91 -0.94 -0.74 -0.8 -0.74 -0.71 -0.26 0.51 0.88 0.86 0.72 0.65 0.59 0.53 0.86 0.86 0.86 0.93 0.93 -0.92 -0.21 -0.2 -0.32 -0.46 -0.78 -0.81 -0.8 -0.41 0.4 -0.06 -0.17 0.08 0.91 -0.94 -0.74 -0.8 -0.74 -0.71 -0.26 0.51 0.88 0.86 0.72 0.65 0.59 0.52 0.89 0.84 0.89 0.91 0.91 -0.92 -0.21 -0.21 -0.33 -0.46 -0.78 -0.81 -0.8 -0.41 0.42 -0.05 -0.16 0.08 0.3 -0.94 -0.75 -0.81 -0.75 -0.71 -0.25 0.51 0.88 0.86 0.72 0.65 0.59
0.75
0.50
0.25
0.00
0.25
0.50
0.75
=1 =5 =10 =15 =20 MSE KL
Emb HS1 HS2 HS3 HS4 HS5 HS6
K1 K2 K3 K4 K5 K6 Q1 Q2 Q3 Q4 Q5 Q6 V1 V2 V3 V4 V5 V6
-0.56
0.7
-0.26 -0.21 -0.12 -0.36 0.11 -0.64 0.79 0.91
0.9 0.93 0.89 -0.95 0.66 0.92
0.8 0.95 0.94 -0.44 0.67 0.84 -0.22 -0.13 0.07
0.91 0.62 0.86 0.87 0.83 0.91 0.95 -0.2 0.89 0.86 0.91 0.92 0.79 -0.71 0.75 0.66 0.63 0.88 0.67 -0.24 0.56 0.89 0.78 0.71 0.65 0.94 0.62 0.81 0.85 0.87 0.86 0.87 0.32 0.88 0.86 0.92 0.91 0.78 0.32 0.63 0.63 0.56 0.85 0.42 -0.19 0.56 0.9 0.81 0.69 0.63 0.95 0.62 0.78 0.82 0.88 0.81 0.71 0.4 0.88 0.87 0.92 0.91 0.8 0.44 0.56 0.62 0.53 0.85 0.36 -0.18 0.6 0.91 0.82 0.68 0.62 0.95 0.62 0.77 0.81 0.88 0.79 0.63 0.42 0.88 0.87 0.93 0.9 0.82 0.49 0.52 0.62 0.52 0.85 0.36 -0.18 0.64 0.91 0.82 0.68 0.62 0.88 0.61 0.8 0.7 0.85 0.82 0.73 0.48 0.87 0.87 0.84 0.56 0.4 0.22 0.89 0.94 0.89 0.89 0.75 -0.18 0.76 0.92 0.87 0.61 0.38
Distance Distance after pair-wise scaled dot-product
Figure 2: Spearman correlation coefficient of the L′sp (τ = 15) with the distance between teacher and student features during pre-training stage of GLMD−vc (T1-S2). Let the intermediate feature be H ∈ Rl×h, and the distance after pair-wise scaled dot-product is calculated by first computing f(H) = HH T
√
h
. HS1, Att1, Q1, K1,
and V1 denote the hidden state, attention scores, query, key, and value of the first layer transformer, respectively. Emb denotes the output of the embedding-layer.

heading: 6 Analysis
text: In this section, we analyze the reasons behind the work of GLMD and the impact of hyperparameters on performance in GLMD.

heading: 6.1 Why Does GLMD Work?
text: Compared to methods using only soft or hard labels, the L′sp in GLMD−vc clearly provides more knowledge, but it is still unclear why the intermediate feature is not necessary. We hypothesize that L′sp reduces inductive bias and allows the model to spontaneously learn intermediate features that should be similar to the teacher. To verify this hypothesis, we calculate the spearman correlation between the distance d(fS(HS), fT (HT )) and L′sp during pre-training stage of GLMD−vc. The red part in Figure 2 shows that as L′sp decreases, not all the distance of features between teacher and student is getting close during distillation so that it
may not be necessary to draw all the intermediate features close as in existed methods, supporting our hypothesis.
We hypothesize that the success of the vocabulary compression strategy is based on the long-tail distribution of tokens, where some low-frequency tokens can still be understood by the language model after being replaced with similar tokens. Figure 3 verifies the long-tail distribution of tokens. The result in row 16 of Table 4 shows that using token mapping for teacher results in a decrease in performance. This verifies that even when some low-frequency tokens are replaced with similar tokens, students can still learn the meaning of these tokens from teachers without token mapping.

heading: 6.2 Hyper-parameter Analysis
text: In Table 5, we analyze the impact of these hyperparameters on performance: τ in L′sp and Lst, batch size in pre-training stage, and rv in GLMD−vc. We find that the temperature hyperparameter (τ) has a significant impact on the performance of L′sp (rows 4-7) but little effect on Lst (rows 8-14). Similarly, in L′sp, we observe that the batch size during the pre-training stage is roughly proportional to performance (rows 15-18). The compression ratio (rv) in our vocabulary compression strategy (rows 21-23) also follows this trend as a higher rv results in more parameters being retained. It is worth noting that the teacher models (T1 and T2) required a batch size of 1024 during the pre-training process, which is significantly larger than the batch size we used in distillation.

heading: 6.3 Limitation
text: Due to limitations in time and computational resources, we limited our experiments to using GLM and SuperGLUE benchmark3. While transformerbased language models and the SuperGLUE benchmark are representative, further validation is necessary when applied to a wider range of models and tasks. Additionally, we found that the performance of GLMD−vc (10B→2B) at 85.28% was marginally lower than that of GLM-2B at 85.91%. However, it’s noteworthy that GLM-2B leverages a substantially greater scale in the pre-training stage with a batch size, iterations, and GPU count of 7168, 17k, and 224 respectively, far exceeding the respective parameters of 64, 15k, and 8 employed by GLMD−vc (10B→2B) in its distillation during the pre-training stage. We plan to further investigate these potential limitations in future work.
3Given the requirement for grid search and seed averaging, we have run over a thousand SuperGLUE averages.

heading: 7 Conclusions
text: In this paper, we introduce a general language model distillation method called GLMD. GLMD has two main advantages: improving distillation performance without relying on intermediate layer features and hard labels and reducing vocabulary parameters without reducing dimensions. We also had two important findings: distillation of intermediate layer features is unnecessary, and a vocabulary compression strategy that reduces the number of tokens is feasible and can be combined with a method that reduces dimensions. In the future, we plan to explore model distillation on a 100B-scale and apply it to more real-world scenarios.
Ethical Statement
This paper aims to compress language models using knowledge distillation methods, and the proposed method do not raise ethical problems or potential biases. All language models, baselines, and datasets used in this work are publicly available and widely used.

heading: Acknowledgements
text: This work is supported by Technology and Innovation Major Project of the Ministry of Science and Technology of China under Grant 2020AAA0108400 and 2020AAA0108402, the Natural Science Foundation of China under Grant No. 61836013, the Major Program of the National Social Science Foundation of China under Grant No. 18ZDA032, and funds from CCFZhipu.AI and Beijing Academy of Artificial Intelligence (BAAI). The GPUs used are sponsored by Zhipu.AI.

heading: B Additional Analysis
text: In this section, we further explore the various factors that influence the performance of the baselines and examine the necessity of intermediate layer features.
B.1 What Factors Affect Performance?
In the implementation of baselines, we discover that certain methods for initializing the student parameters led to a decrease in performance, surpassing even the advantages brought about by the method innovation. Specifically, these include the following three ways: (1) using truncated teacher
parameters in the case of different dimensions of teacher and student hidden layers. Many methods that do not distill in the pre-training stage will use the parameters of the first few layers of the fine-tuned teacher as student parameters in a taskspecific stage. In the case of different dimensions of teacher and student hidden layers, we can only truncate some parameters per layer. As shown in Table 3, Universal-KD (Wu et al., 2021b) in T1S2 performs much better than T1-S2 and T2-S2, TAKD (Mirzadeh et al., 2020) and DGKD (Son et al., 2021) also performs badly due to this reason. (2) Using less data to pre-train a student model as initialization. To ensure fairness, regardless of whether distillation is used, we set the batch size of all methods in the pre-training stage to 64, which is equivalent to only using one-sixteenth of the full data (batch size=1024). Some methods that use a pre-trained student as initialization for a taskspecific stage may be affected by this, for example, PD (Turc et al., 2019), SID (Aguilar et al., 2020), LRC-BERT (Fu et al., 2021), Annealing-KD (Jafari et al., 2021), Continuation-KD (Jafari et al., 2022), and CKD (Park et al., 2021). (3) Randomly initializing the parameters of the student model. As can be seen from Table 3, KD (Hinton et al., 2015) using random initialization are obviously inferior to PD (Turc et al., 2019) using pre-trained
students and soft labels. The above analysis demonstrates that methods without pre-training distillation are sensitive to the initialization of the student’s parameters. To achieve optimal performance, methods based on truncating teacher parameters require the hidden dimensions of the teacher and student to be identical or else, other methods would require a significant cost in pre-training a student model. Therefore, utilizing a subset of the corpus for knowledge distillation during the pre-training stage is a more favorable option.
B.2 Why are Intermediate Layers not Necessary? In Section 6.1, we have verified that the L′sp in GLMD−vc can enable the model to spontaneously learn intermediate layer features that should be similar to those of the teacher. We further validate in Figure 5 that training with a loss function focused on the intermediate layer features does not lead to a reduction in L′sp and cannot even lower the perplexity (PPL) of language modeling. In the process of distillation using GLMD−vc and TinyBERT methods, we have quantified the spearman correlation between the distance of student and teacher features, including the perplexity of the student model on the validation set, and the loss function values. We observe that there is no correlation between
the loss function values of TinyBERT and L′sp, nor with the perplexity of the validation set. This suggests that we may not require a significant inductive bias towards the intermediate layer features.

heading: C Detailed Results
text: Due to space constraints, we do not present results for all datasets in the SuperGLUE benchmark in the main text but only show the averages. Table 9 shows the results for all methods on each dataset in the SuperGLUE benchmark, rounded to two decimal places.
ACL 2023 Responsible NLP Checklist

heading: A For every submission:
text: 3 A1. Did you describe the limitations of your work?
6.3 Limitation
3 A2. Did you discuss any potential risks of your work? Ethical Statement
3 A3. Do the abstract and introduction summarize the paper’s main claims? Abstract and 1 Introduction
7 A4. Have you used AI writing assistants when working on this paper? Left blank.
B 7 Did you use or create scientific artifacts? Left blank.
B1. Did you cite the creators of artifacts you used? No response.
B2. Did you discuss the license or terms for use and / or distribution of any artifacts? No response.
B3. Did you discuss if your use of existing artifact(s) was consistent with their intended use, provided that it was specified? For the artifacts you create, do you specify intended use and whether that is compatible with the original access conditions (in particular, derivatives of data accessed for research purposes should not be used outside of research contexts)? No response.
B4. Did you discuss the steps taken to check whether the data that was collected / used contains any information that names or uniquely identifies individual people or offensive content, and the steps taken to protect / anonymize it? No response.
B5. Did you provide documentation of the artifacts, e.g., coverage of domains, languages, and linguistic phenomena, demographic groups represented, etc.? No response.
B6. Did you report relevant statistics like the number of examples, details of train / test / dev splits, etc. for the data that you used / created? Even for commonly-used benchmark datasets, include the number of examples in train / validation / test splits, as these provide necessary context for a reader to understand experimental results. For example, small differences in accuracy on large test sets may be significant, while on small test sets they may not be. No response.
C 3 Did you run computational experiments? 5 Experiments
3 C1. Did you report the number of parameters in the models used, the total computational budget (e.g., GPU hours), and computing infrastructure used? 5 Experiments
The Responsible NLP Checklist used at ACL 2023 is adopted from NAACL 2022, with the addition of a question on AI writing assistance.
3 C2. Did you discuss the experimental setup, including hyperparameter search and best-found hyperparameter values? 5.1 Experimental Setup and A.2 Hyperparameters
3 C3. Did you report descriptive statistics about your results (e.g., error bars around results, summary statistics from sets of experiments), and is it transparent whether you are reporting the max, mean, etc. or just a single run? 5.1 Experimental Setup
3 C4. If you used existing packages (e.g., for preprocessing, for normalization, or for evaluation), did you report the implementation, model, and parameter settings used (e.g., NLTK, Spacy, ROUGE, etc.)? 5.1 Experimental Setup
D 7 Did you use human annotators (e.g., crowdworkers) or research with human participants? Left blank.
D1. Did you report the full text of instructions given to participants, including e.g., screenshots, disclaimers of any risks to participants or annotators, etc.? No response.
D2. Did you report information about how you recruited (e.g., crowdsourcing platform, students) and paid participants, and discuss if such payment is adequate given the participants’ demographic (e.g., country of residence)? No response.
D3. Did you discuss whether and how consent was obtained from people whose data you’re using/curating? For example, if you collected data via crowdsourcing, did your instructions to crowdworkers explain how the data would be used? No response.
D4. Was the data collection protocol approved (or determined exempt) by an ethics review board? No response.
D5. Did you report the basic demographic and geographic characteristics of the annotator population that is the source of the data? No response.

heading: Future Work
text: In the future, we plan to explore model distillation on a 100B-scale and apply it to more real-world scenarios