HomoDistil: Homotopic Task-Agnostic Distillation of Pre-trained Transformers
Authors: Chen Liang
Haoming Jiang
Zheng Li
Xianfeng Tang
Bing Yin
Tuo Zhao
Venue: ICLR 2023 poster (2023)
Keywords: Knowledge Distillation
Structured Pruning
Pre-trained Transformer Language Models
Model Compression

Abstract: Knowledge distillation has been shown to be a powerful model compression approach to facilitate the deployment of pre-trained language models in practice. This paper focuses on task-agnostic distillation. It produces a compact pre-trained model that can be easily fine-tuned on various tasks with small computational costs and memory footprints. Despite the practical benefits, task-agnostic distillation is challenging. Since the teacher model has a significantly larger capacity and stronger representation power than the student model, it is very difficult for the student to produce predictions that match the teacher's over a massive amount of open-domain training data. Such a large prediction discrepancy often diminishes the benefits of knowledge distillation. To address this challenge, we propose Homotopic Distillation (HomoDistil), a novel task-agnostic distillation approach equipped with iterative pruning. Specifically, we initialize the student model from the teacher model, and iteratively prune the student's neurons until the target width is reached. Such an approach maintains a small discrepancy between the teacher's and student's predictions throughout the distillation process, which ensures the effectiveness of knowledge transfer. Extensive experiments demonstrate that HomoDistil achieves significant improvements on existing baselines. Our codes will be released.

Published as a conference paper at ICLR 2023
HOMODISTIL: HOMOTOPIC TASK- DISTIL- LATION OF PRE-TRAINED TRANSFORMERS
Chen Liang"; Haoming Jiang Zheng Li", Xianfeng Tang", Bin Yin & Tuo Zhao*
"Georgia Institute of Technology, 9 Amazon
{cliang73, tourzhao @ggttchhh...................hh.. edu,
{jhaoming: ng, amzzhe, x xi fantt,, ,, a e @amazon,aaaammcccmmmmmmmmm com
ABSTRACT
Knowledge distillation has been shown be powerful model compression ap- proach to facilitate the deployment of pre-trained language models in practice. This paper focuses task-agnostic distillation. It produces compact pre-trained model that can be easily fine-tuned on various tasks with small computational costs and memory footprints. Despite the practical benefits, task-agnostic distil- lation is challenging. Since the teacher model has a significantly larger capacity and stronger representation power than the student model, it is very difficult for the student to produce predictions that match the teacher's over massive amount of open-domain training data. Such a large prediction discrepancy often dimin- ishes the benefits of knowledge distillation. To address this challenge, we propose Homotopic Distillation (HomoDistil), a novel task- aus--agnsttic distillation approach equipped with iterative pruning. Specifically, we initialize the student model from the teacher model, and iteratively prune the student's 's neurons until the target width is reached. Such approach maintains small discrepancy between the teacher's s and student's predictions throughout the distillation process, which ensures the effectiveness of knowledge transfer. Extensive experiments demonstrate that Ho- moDistil achieves significant improvements on existing baselines.
INTRODUCTION
Pre-trained language models have demonstrated powerful generalizability in various downstream applications (Wang et al.) [0188] Rajpurkar et al., (016a)) However. the number of parameters in such models has grown over hundreds of millions (Devlin et al., 2018 Raffel et al.] 2019: Brown et al. 2020). This poses a significant challenge to deploying such models in applications with latency and storage requirements.
Knowledge distillation (Hinton et al. 2015) has been shown to be powerful technique to compress a large model (i.e., teacher model) into a small one (i.e., student model) with acceptable performance degradation. It transfers knowledge from the teacher model to the student model through regulariz- ing the consistency between their output predictions. In language models, many efforts have been devoted task-specific knowledge distillation (Tang et al. 2019 Ture et al. 2019: (un al.) 2019 Aguilar et al., 2020). In this case, a large pre-trained model is first fine-tuned on a downstream task, and then serves the teacher to distill student during fine-tuning. However, task-specific distillation is computational costly because switching to a new task always requires the training of task-specific teacher. Therefore, recent research has started to pay more attention to task cass-agnostcc
distillation (Sanh et al. 2019 Sun et al. 2020 Jiao et al. 2019 Wang et al., 2020b Khanuja et al.) 2021) ) where a student distilled from a teacher pre-trained on open-domain data
2021 Chen and can be efficiently fine-tuned on various downstream tasks.
Despite the practical benefits, task task agnostic distillation is challenging. The teacher model has significantly larger capacity and much stronger representation power than the student model. As a result, it very difficult for the student model produce predictions that match the teacher's
Work done while interning Amazon.

---

Published as a conference paper at ICLR 2023
DKL
Teacher
Criadion
Catta
Effective Distillation Region
HomoDistil: Prume Distillation
Daidden Castn
HomoDistil Student
Distilled Student
Teacher
Student in HomoDistil
Studeni Inittalization
Stundard Distillation
Figure 1: Left: In Histli the student initialized from the teacher and iteratively pruned through the distillation process. The widths of rectangles represent the widths of layers. The depth of color represents the sufficiency of training. Right: An illustrative comparison of the student's optimization trajectory in HomoDistil and standard distillation. We define the region where the prediction discrepancy is sufficiently small such that the distillation is effective as the Effective Dis- tillation Region. HomoDistil, as the student is initialized with the teacher and is able to maintain this small discrepancy, the trajectory consistently lies in the region. In standard distillation, as the student initialized with much smaller capacity than the teacher's, 's the distillation is ineffective at the early stage of training.
over massive amount of open-domain training data, especially when the student model is not well- initialized. Such a large prediction discrepancy eventually diminishes the benefits of distillation (Tin
al. 2019 Cho & Hariharan] 2019) Mirzadeh al. 2020 Guo al.] 2020 Li et al. 2021). To
reduce this discrepancy, recent research has proposed to better initialize the student model from
subset of the teacher's layers (Sanh et al.) 2019) Jiao et al. 2019. Wang et al. 2020b). However, selecting such subset requires extensive tuning.
To address this challenge, we propose Homotopic Distillation (HomoDistil), a novel task-agnostic distillation approach equipped with iterative pruning. As illustrated in Figure [ we initialize the student model from the teacher model. This ensures a small prediction discrepancy in the early prune set of least important neurons, which
stage of distillation. At each training iteration, leads to the least increment in loss due to its removal, from the remaining neurons. This ensures the prediction discrepancy only increases by small amount. Simultaneously, we distill the pruned student, such that the small discrepancy can be further reduced. We then repeat such a procedure in each iteration maintain the small discrepancy through training, which encourages an effective knowledge transfer.
We conduct extensive experiments demonstrate the effectiveness of HomoDistil in task-agnostic distillation on BERT models. In particular, HomoBERT distilled from a BERT-base teacher (109M) achieves the state-of-the-ar fine-tuning performance on the GLUE benchmark (Wang et 2018) and SQuAD v1.1/2.0 (Rajpurkar et al. 2016aa 2018) at multiple parameter scales (e.g., 65M and 10 20M). Extensive analysis corroborates that HomoDistil maintains small prediction discrep- ancy through training and produces better-generalized student model.
2
PRELIMINARY
2.1 TRANSFORMER-BANE LANGUAGE MODELS
Transformer architecture has been widely adopted to train large neural language models (Vaswani et al. 2017 Devlin et al.) 2018 Radford et al. 2019 He et al.] 2021). It contains multiple identically constructed layers. Each layer has multi- head self-attention mechanism and a two-layer feed- forward neural network. We use f(; () to denote a Transformer-basee model f parameterized by 0, where f is a mapping from the input sample space X' to the output prediction space. We define the loss function C(0) Ez-x [=(; 0))] where is the task loss
2.2 TRANSFORMER DISTILLATION
Knowledge Distillation trains a small model (i.e., student model) to match the output predictions of large and well-trained model (i.e., teacher model) by penalizing their output discrepancy. Specif-
For notational simplicity, we will omit throughout the rest of the paper.
2

---

Published as a conference paper at ICLR 2023
ically, denote the teacher model as f(())) and the student model fa,),,, and consider the following optimization problem:
min [(,)) + (R.. 6,
(1)
where (0,. 0.) is the KL-Divergence between the probability distributions over their output predictions, i.c., (f))(()))l))())))
Transformer Distillation. In large Transformer-based models, distilling knowledge from only the output predictions neglects the rich semantic and syntactic knowledge in the intermediate layers. To leverage such knowledge, researchers have further matched the hidden representations, attention scores and attention value relations at all layers the teacher and the student (Romero et al. 2014
Sun al. 2019: 2020
Jiao al. 2019
Hou et al. 2020 Wang et al. 2020b
2.3 TRANSFORMER PRUNING
Pruning is powerful compression approach which removes redundant parameters without signifi- cantly deteriorating the full model performance Han et al. (20 5b a): Paganini & Forde (2020); Zhu
& Guupta 2017; Renda et al. (2020); Zafrir et al. [2021) Liang et al. (2021).
Importance Score. To identify the redundant parameters, researchers estimate the importance of each parameter based on some scoring metries. A commonly used scoring metric is the sensitivity of parameters (Molchanov et al., 2017 2019] Theis et al. 2018 Lee et al. 2019; Ding et al. 2019; Xiao al. 2019). It essentially approximates the change the loss magnitude when this parameter is completely zeroed-out (LeCun et al., 1990, Mozer & Smolensky 1989). Specifically, we denote 0 10. ..... 0,] E RJ, where 8j IR for = I..... denotes each parameter. We further define J [0. ..... 0, 0, O.... 0) E R. Then define the sensitivity score S E R'', where Sy E R
computes the score of 8j as
S, 5
(2)
This definition is derived from the first-order Taylor expansion of L(-) with respect 0g 0. Specifically, Sj approximates the absolute change of the loss given the removal of 0y:
(((( [(0) L(B ((0-0)--) 0g.-))-
The parameters with high sensitivity are of high importance and should be kept (Lubana & Dick, 2020). Parameters with low sensitivity are considered redundant, and can be safely pruned with only marginal influence on the model loss. Other importance scoring metrics include the magnitude of parameters |Han al. (2015b) and the variants of sensitivity, e.g., movement score (Sanh al., 2020), sensitivity score with uncertainty (Zhang et al. 2022). and second-order expansion of Eq (LeCun et al. 1990).
Iterative Pruning gradually zeroes out the least important parameters throughout the training pro- cess. Specifically, given gradient updated model ((t) at the t-th training iteration, iterative prun- 2 then compute a binary mask
ing methods first compute the importance score 5(() following Eq
(*) E R as
M - if (()))) is in the top (()) of S(t)
Vjj I...,,
otherwise.
where (1) E (0. 1) is the scheduled sparsity at the t-th iteration determined by a monotonically decreasing function of L. Then the model is pruned as M(0) @ p(t), where @ denotes the Hadamard product. Such a procedure is repeated through training.
Structured Pruning. Pruning the model in the unit of single parameter leads to highly sparse subnetwork. However, the storage and computation sparse matrices are not often optimized on commonly used computational hardware. Structured pruning resolves this issue by pruning the model in the unit of a structure, e.g., a neuron, an attention head, or a cord-forward layer (Wang et al., 2019| Michel et al.] 2019 Liang al.) 2021; Hou et al. 2020: Lagunas et al. (021). To estimate the importance score structure, existing works compute the expected sensitivity with respect the structure's output (Michel et al. 2019 Liang et al. 2021 Kim & Awadalla 2020).
3

---

Published as a conference paper at ICLR 2023
3 METHOD
We introduce Homotopic Distillation, as illustrated in F Specifically, we initialize the student model from the teacher model. At each iteration, prune the least important neurons from the student and distill the pruned student. We repeat such a procedure throughout the training process.
Task-Agnostic Distillation. We consider the following losses to optimize the student model: 1) The knowledge distillation loss as defined in Eq I In task-agnostic distillation, L is the loss for continual pre-training of the student model on the open- open-domain data, e.g., the masked language modeling loss for BERT, LMLM- 2) The Transformer distillation losses. Specifically, we penalize the discrepancy between the teacher's and the student's hidden representations at both the intermediate and embedding layers, and the attention scores at the intermediate layers. We denote the hidden representations at the k-th intermediate layer of the teacher and the student as H E R xd, and H E [R/=|xd,, where d, and d, denote the hidden dimension and |a]] denotes the sequence length. The distillation loss of the hidden representations at the intermediate layers is defined as:
Luida 01,
Σ MSE(HI) H. W
Here MSE(;)) is the mean-squared error, and Whi E Rd do is a randomly initialized and learn- able linear projection that projects H. into the same space as H.S. Similarly, the distillation loss of the hidden representations at the embedding layer is defined
Lemb (0.. 0₂) = MSE F,, E, Wemb);
where Eg E Rlxx and E, Rlalxd, are the hidden representations at the embedding layer and Wemb E Rdd xd, for dimension matching. Finally, the attention distillation loss is defined as
K
Latta (0,, 01) = Σ (A A
where A$ RIa|x|= and A R|=**|= are the attention of heads at the k-th layer. These transformer distillation losses aim to capture the rich semantic and syntactic knowledge from the teacher's layers and improve the generalization performance of the student. In summary, the student optimized based on the weighted sum of all losses, i.e.,
matrices averaged by the number
Liotal .MLM + 01 PKL + az agyeemb + agLattns
where (1, 022 03, 04 are hyper-parameters.
Iterative Neuron Pruning. We initialize the student model from pre-trained teacher model
0 O2-
At the f-th training iteration, update the student model based on Liotal defined in Eq Jsing SGD-type algorithm, e.g.,
( - ((t-)) - Vot (,)))
where 7 is the step size. Then compute the importance score for all parameters following Eq 2
5iD = T., Canna 0:)l Vj I,
For any weight matrix ((t) E a xd, the student model, we denote its corresponding impor- tance score s(t) xd, We then define the importance score for individual columns as (i(1) Rd,, where
(1 = || r.lll
=i= L,.,,d,
Notice that the score is computed based Liotals which consists of both the distillation and training losses. This is to ensure that we only prune the columns whose removal would lead to the least increment both the prediction discrepancy and the training loss.
4

---

an English Wikipedia corpus containing 2500M words, and Published as a conference paper at ICLR 2023
We then compute the binary mask M E Rdd associated with the weight matrix following Eq as
W { if NW is in the top r() of (o(0)
W. Vi 1, dg.
(8)
otherwise,
where r(() is the scheduled sparsity determined by commonly used cubically decreasing function Zhu & Gupta 2017 Sanh et al. 2020 Zafrir et al. 2021):
051<4 4st<t/
,00)
ry + (1- ty
1-4
rf
st
Here ry is the final sparsity, T is the number of total training iterations and 0 S < ty S T hyper-parameters. Such schedule ensures that the sparsity is slowly increasing and the columns gradually pruned. This prevents a sudden drop in the student's prediction performance, which effectively controls the expansion of prediction discrepancy.
Finally, prune the weight matrix ((()) MW We also prune the corresponding rows of the next weight matrix in the forward computation, including {WA, pruning procedure is applied to all weight matrices in the model. The complete algorithm is shown in Alg.
hidn hidn /== and Wemb- The same
Algorithm HomoDistil: Homotopic Distillation
1: Input: 0;: the teacher model. T, 1, f F 0n, 02, 03, Oua: Hyper- Plyper-parameters
2: Output: o!T")
3: (00) = -,,.
4: fort
do 5: Compute loss Liotal following 5q
6: 0 + 0
1) Ltotal-
7: Compute importance score 5(t) following Eq
for all ( E 0²²) do
8:
9: Compute importance score for individual columns, N1t following Eq
10: Compute binary mask M! following Eq
11: ((t) + W(()) @ (
end for
13:
end for
Why do we impose sparsity requirements on individual weight matrices? Traditional pruning imposes requirements on the global sparsity the model instead of the local sparsity of the indi- vidual matrices. As a result, some matrices have much larger widths than the others. These wide matrices can be the memory bottlenecks for commonly used computational hardware. Furthermore, it requires re-configurations of the pre-defined model architectures in deep learning software pack- ages to achieve the desired inference speedup. In contrast, controlling the local sparsity more friendly to both hardware and software.
4 EXPERIMENTS
We evaluate HomoDistil on BERT-base (Devlin et al. 2018 on natural language understanding (NLU) and question answering tasks.
4.1 DATA
Continual Pre-training, We distill the student using the open-domain corpus for BERT pre-training (Devlin et al. 2018), i.e., Wikipedia ?

---

Published as a conference paper at ICLR 2023
Toronto BookCorpus (Zhu et al. 2013), containing 800M words. We clean the corpus by removing tables, lists and references following BERT. We then pre-process the cleaned corpus by concatenat- ing all sentences in paragraph and truncating the concatenated passage by length of 128 following TinyBERT (Jiao et al. 2019). We tokenize the corpus with the vocabulary of BERT (30k).
We fine-tune the student model on both NLU and question answering tasks. For NLU tasks, we adopt the commonly used General Language Understanding Evaluation (GLUE) bench- mark (Wang et al. 2018), which contains nine tasks, e.g., textual entailment, semantic similarity, etc. For question answering tasks, we adopt the SQuAD v1.1 and v2.0 (Rajpurkar et al. 201 2018 Details about the datasets are deferred to Appendix A.1
Fine-tuning.
4.2
MODEL
We evaluate HomoDistil on pre-trained BERT-base (Devlin et al. 2018), which contains 12 Trans- former layers with hidden dimension 768. BERT-base is pre-trained with masked language modeling and next sentence prediction tasks on Wikipedia and Toronto BookCorpus (16GB). We use BERT- base as the teacher model and as the initialization of the student model. We produce multiple student models at several sparsity ratios. Table Tists the architectures of the teacher and the student models.
Table Architectures of the teacher and the student models.
Params (million)
Model
Embedding
Backbone
Total
hidn
dff
BEKT-base (Teacher)
23.4
85,5
109
768
3072
HomoBERRTbbase
17.6
47.8
65
576
2304
HomoBERT
7.8
9.4
17.3
256
1024
MomooBEKT-ssmall
7.3
8.3
15.6
240
960
HonoBERT-inn
7.2
6.8
14.5
224
896
4.3 BASELINES
We compare HomoDistil with the state-of-the-art task-agnostic distillation baselines These meth- ods initialize the student directly as the target size and fix its size during distillation. For example, to obtain a shallow model, the student is often initialized from a subset of teacher's 's layers.
DistiBBERT (Sanh al. 2019) considers the vanilla distillation by penalizing the final layer predic- tion discrepancy using Eq
BERTGERGGD (General Distillation) (Jiao et al.
2019) extends DistilBERT by exploiting the
knowledge in the intermediate Transformer layers using 5q
MiniLMM (Wang et al.) 20206) penalizes the discrepancy between the queries-keys scaled dot prod- uet and values- values scaled dot product in the final layer self-attention module. Mv2 (Wang et al.) 2020a) extends MiniLM by encouraging the student to mimic the attention head relations of the teacher.
4.4 IMPLEMENTATIONS DETAILS
Continual Pre-training. For all experiments, we use a max sequence length of 128 and a batch size of 4k. We train the student model for T 28k steps (3 epochs). We use Adam (Kingmal 1 x 10-6, We use a learning rate of
& Ba 2014) as the optimizer with 3 = (0.9, 0.999), 3 x 10 for HomoBERT-base and 6 10-4 for KomooBRRT-malllxssalll
We adopt linear decay learning rate schedule with a warmup ratio of 0.1. For distillation, we share all weights of Whida and (W& [ - We set a1, 022 Q3, 04 to be for all experiments. For importance score computation, we select neurons based on the exponential moving average of the importance score for stability. For pruning schedule, we set the initial iteration 4 as and select the final iteration I from (0.5, 0.7, 0.9) x T. Full implementation details are deferred to Appendix A.2
Fine-tuning. We drop the masked language modeling prediction head and Whidn and {W 1K from the continual pre-training stage, and randomly initialize task-specific classification head for
We mainly compare with baselines that use BERT-base as the teacher model for fair comparison. We also present comprehensive comparison with task-specific distillation haselines in Appendix A.4

---

Published as a conference paper at ICLR 2023
the student model. For NLU tasks, select the training epochs from {3. 6). batch size from {16, 32) and learning rate from {2, 3, 5, 6, 7 x 10 For RTE, MRPC and STS-B, we initialize the student from MNLI-finetunna
student to further improve the performance for all baselines. For question answering tasks, we fine-tune the student for epochs with a batch size of 12, and adopt a learning rate of 1 10-1 1 For all tasks,
Adam the optimizer with with 3 = (0.9, 0.999), X 10-6. Full implementation details are deferred to Appendix A.3
4.5
MAIN RESULTS
Table 2 show the fine-tuning results of HomoDistil on the GLUE development set. We report the median over five random seeds for all experiments in this paper $ HomoBERT-base consistently outperforms existing state-of-the-art baselines over six out of eight tasks, and achieves significant gains on MNLI, SST-22 and CoLA. The margins of gains become much more prominent for students with 10 ~ 20M parameters: HomoBERT-iny (14.1M) significantly outperforms TinyBERT4x00 (14.5M) by 3.3 points in terms of task-average score, and outperforms BERT-small, which is twice of the scale, by 1.0 point.
Table show the fine-tuning results of HomoDistil SQuAD v1.1/v2.0. All HomoBERT stu- dents outperform the best baseline, MiniLMg (17.3M), by over 3 points of margin on SQuAD v2.0. Especially, HomoBERT-xsmall (15.6M) obtains 3.8 points of gain.
Table 2: The accuracy of fine-tuning distilled BERT models on GLUE development set. The results of MiniLMa,e are reported from (Wang et al., 2020b). The rest are fine-tuned from the officially
released checkpoints (Devlin et al. 2018 Sanh et al. 2019: Wang et al., 2020a Jiao et al. 2019).
Params (million)
MNLI
QQP Acc/F1
QNLI
SST-2
CoLA
RTE
MRPC
STS-B P/S
Avg Score
Model
Acc
Acc
Acc/F1
BERT-base (Teacher)
109
84.5/84.6
91.1/88.1
91.2
92.9
58.7
79.8
89.5/92.4
89.3/89.2
84.6
DistiBBRRT
66
82.4/82.5
90.4/87.1
89.2
90.9
53.5
75.5
86.5/90.5
87.9/87.8
82.1
TinD
66
83.5/-
90.6/-
90.5
91.6
42.8
77.3
88.5/91.6
89.0/88.9
81.7
MiniLM,
66
84.0/-
91.0/-
91.0
92.0
49.2
-/-
MiniLMv2,
66
84.0/-
91.1/-
90.8
92.4
52.5
78.0
88.7/92.0
89.3/89.2
83.4
HomoBERT-buse
65
84,2/84.3
91.2/87.9
92.7
55.9
77.6
89.0/91.9
89.5/89.2
83.8
BERT-small
28.6
78.8/78.9
89 99.9/86.5
87.0
88.2
36.1
70.8
85.8/90.1
87.7/87.7
78.0
TinyBERT J
17.0
77.4/-
-f-
88.4
-/-
MiniLM,
17.0
78.8/-
88.8/85.0
84.7
89.3
15.8
66.4
81.9/88.2
85,4/85.5
73.9
Tiny T
14.5
80.4/80.9
88.7/85.3
85.7
18.6
71.1
84.6/89.1
87.0/87.2
75.7
HomoBtiRT-tny
14.1
81, 81.2/81.3
89.9 89.9/86.6
87.8
90.1
37.0
70.8
87.3/90.7
87.6/87.5 5
79.0
HomoBERT-Asmall
15.6
81.5/81.5
90,0/86.7 7
88.0
90.3
40.8
71.5
87.7/91.0
88.3/88.0
79.7
HomoBB-KTsmall
17.3
81.8/81.
90. 1/86.9
88.5
91.1
42.1
72.6
88.0/91.4
88.3/88.1
80.3
Table The accuracy of fine-tuning distilled models on SQuAD v 1/2.0 validation set. The results of TinyBERT 3-D and MiniLM, are reported from WWang et al., 20206). The rest are fine-tuned from the officially released checkpoints (Devlin et al. 2018
Jiao et al.) 2019).
Model
Params (million)
SQuAD v1.1
SQuAD v2.0
Avg
EM/FI
EM/FI
BERT-buse (Teacher)
109
81.7/88.9
73.4/76.7
82.8
BBRT-small
28.6
72.5/81.5
61.3/64.8
73.2
17.0
-f-
-163.6
MiniLM;
17.0
-f-
-166.2
TinyBERT GGD
14.5
60,8/72.3
58.9/63.3
67.8
HomoBERT-ting
14.1
75.5/84.1
66.1/6
76.8
HomoBERT-xsae
15.6
76,2/84.5
66.5/70.0
77.2
HomoBERT-small
17.3
76.5/84.8
66.6/69.8
77.3
5 ANALYSIS
We verify that HomoDistil maintains small prediction discrepancy throughout the distillation pro- cess, leading to a better-generalin student model.
*The standard deviations are reported in Appendix A.7

---

Published as a conference paper at ICLR 2023
5.1 HOMODISTIL MAINTAINS SMALL PREDICTION DISCREPANCY
Figure shows the prediction discrepancy, DKL. under different schedules of sparsity throughout the distillation process. When the student is directly initialized with single-shot pruned subnetwork at the target sparsity (i.e., / 0), the initial prediction discrepancy is large. In contrast, when the student initialized with the full model and is iteratively pruned through longer iterations (i.e., Ł 0.5I, 0.7I' and 0.91"), the initial discrepancy is small. The discrepancy then gradually increases due to pruning, but the increment remains small due to distillation.
HomoBERT BERbbbase
HomoBERT
0.7
1.75
8
1.50
0.6
0.57
1.25
0.77
0.97
1.00
0.75
0.50
10
20
10
20
Training Steps (k)
Training Steps (k)
Figure 2:
The prediction discrepancy during the distillation of HomoBERT models under different
schedules of sparsity.
Figure shows the accuracy of task-specific fine-tuning of the student distilled with different sched- ules of sparsity. The student that is initialized with the full model and is pruned iteratively achieves a significantly better generalization performance on the downstream tasks than the one initialized to be the target-size subnetwork.
MNLI
SST-2
RTE
74
Accuracy (%)
82-
81,7
81.6
81,8
91
90.7
91,1
91,1
2
72
72
80-
80.0
70.8
70
0.0
0.5
0.7
0.9
0.0
0.5
0.7
0.9
0.0
0.5
0.7
0.9
ty
ly
Figure 3: The accuracy of fine-tuning HomoBERT-smail distilled with different schedules of sparsity on the development of GLUE benchmark.
5.2 DISTILLATION BENEFITS ITERATIVE PRUNING
Taalee the student trained with and without distillation losses (i.e., Ltotal defined in Eq and LMLM only). The task-specific fine-tuning performance of the student trained with distillation losses consistently outperforms the one without distillation losses over multiple model scales. This suggests that teacher's 's knowledge essential to recover the performance degradation due pruning, and minimizing distillation loss is an important criteria to select important neurons.
5.3
IMPORTANCE METRIC MATTERS
Table investigates the student performance under different importance metrics: 1) Magnitude Pruning (Han et al. 2015b), where S, = O Movement Pruning (Sanh et al. 2020). where Ve 3) ZZhng et al. 2022 Sj =L; U,,. where I, is the sensitivity score as defined in Eq 2 and U, is the uncertainty estimation of Iy. For all methods, we use the exponential moving average of score for stability. Using sensitivity and PLATON as the importance score significantly outperforms the baseline. In contrast, the weight magnitude, which may not correctly quantify the neuron's contribution the loss in the large and complex models, achieves
S, =
8

---

Published as a conference paper at ICLR 2023
Table 4: The accuracy of fine-tuning HomoBERT models trained with and without distillation losses (" ("Lotaal" and "Lmrm") on the development set GLUE benchmark.
Params (million)
MNLI
SST-2
RIE Acc
Avg Score
Importance Score
Objective
BERT-basc (Teacher)
109
84.5/84.6
92.9
79.8
85.7
Listal
84.2/84.3
77.6
84.8
HomoBERTTbase
65
ΣxTM
82.8/83.0
91.6
75.1
83.2
Luast
81.8/818
91.1
72.6
81.8
HomoBERTTssail
79.3/80.1
88.5
71.8
79.9
Diast
81.2/81.3
90.1
70.8
80.7
HomoBB-TTtiny
14.5
DAMM
78.7/79.5
69.7
78.7
only comparable performance to the baseline. Movement pruning, which is mainly designed for task-specific fine-tuning, diverges.
Table 5: The accuracy of fine-tuning HomoBERT-smal pruned under different importance metrics on the development set of GLUE benchmark.
Params (million)
MNLI Acc
SST-2
RTE Acc
Avg Score
Importance Score
Acc
BERT-base (Teacher)
109
84.5/84.6
92.9
79.8
85.7
TinyBERTxxa 4x32-GD
17.0
80.4/80.9
89.7
71.1
80.4
Magnitude Han al. 2015b
17.3
79.7/80.4
90.3
70.4
80.1
Movement Sanh et
17.3
Does converge
Sensitivity
1990
17.3
81.8/818
91.1
72.6
81.8
PLATON Zhang 2022
17.3
81.6/81.9
90.6
73.6
81.9
6 DISCUSSION
Combining pruning and distillation. While we are the first work to combine pruning with dis- tillation task- ausk-agnostic setting, there have been similar explorations in task-specific setting, One stream of explorations first prune the model to the target size and then distill the subnetwork (Hou) al. 2020 Lagunas et al., 202T). In this case, pruning solely serves as an architecture selection strategy independent of distillation. Another stream simultaneously prunes and distills the model (Xu et al. 2021 Xia et al. 2022). which is more comparable to ours. The main differences are that they do not initialize the student with the teacher and often prune at a large granularity, e.g., Trans- former layer. In task-agnostic setting, however, an undesirable initialization and a large granularity will induce huge discrepancy, which is difficult to minimize large amount of open-domain data. Furthermore, after each layer pruning, the remaining layers need to match a different set of teacher layers ensure the learning of comprehensive knowledge. However, suddenly switching the layer to learn from can be difficult on large amount of open-domain data. How to prune the student's height in task-agnostic setting remains an interesting open problem. A comprehensive comparison of these methods is deferred to Appendix A.5
Resolving prediction discrepancy. Recent research has shown that distillation from large teacher
to a small student has only marginal benefits (Jin et al. 2019 Cho & Hariharan 2019), mainly due the large prediction discrepancy (Guo et 20200). Traditional solutions have resorted to
introducing auxiliary teacher assistant models (Mirzadeh et al. 2020: [Rezaghlizzadhh et al. 2021
Li al. 202T), but training and storing auxiliary models can be memory and computational costly.
CONCLUSION
We propose novel task-agnostic distillation approach equipped with iterative pruning HomoDis- til. We demonstrate that HomoDistil can maintain a small prediction discrepancy and can achieve promising benefits over existing task-agnostic distillation baselines.

---

Published as a conference paper at ICLR 2023
REFERENCES
Gustavo Aguilar, Yuan Ling, Yu Zhang, Benjamin Yao, Xing Fan, and Chenlei Guo. Knowledge distillation from internal representations. Proceedings of the AAAI Conference on Artificial Intelligence, volume 34, pp. 7350-7357, 2020.
Roy Bar-Haim, Ido Dagan, Bill Dolan, Lisa Ferro, and Danilo Giampiccolo. The second PASCAL recognising textual entailment challenge. In Proceedings of the Second PASCAL Challenges Workshop on Recognising Textual Entailment, 01 2006.
Luisa Bentivogli, Ido Dagan, Hoa Trang Dang, Danilo Giampiccolo, and Bernardo Magnini. The fifth pascal recognizing textual entailment challenge. In In Proc Text Analysis Conference (TAC'09), 2009.
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. arXiv preprint 211.:::0005.11165. 2020.
Daniel Cer, Mona Diab, Eneko Agire, lñigo Lopez-Gazpin and Lucia Specia. Semeval-200 2017 task I: Semantic textual similarity multilingual and crosslingual focused evaluation. In Proceedings of the 11th International Workshop on Semantic Evaluation (SemEval-2017) pp. 1-14, 2017.
Cheng Chen, Yichun Yin, Lifeng Shang, Zhi Wang, Xin Jiang, Xiao Chen, and Qun Liu. Extract then distill: Efficient and effective task-agnostic bert distillation. arXiv preprint arXiv:2104. 1928, 2021.
Jang Hyun Cho and Bharath Hariharan. On the efficacy of knowledge distillation. In Proceedings of the JEEE/CVF International Conference on Computer Vision, pp. 4794-4802, 2019.
Ido Dagan, Oren Glickman, and Bernardo Magnini. The pascal recognising textual entailment chal- lenge. In Proceedings of the First International Conference on Machine Learning Challenges: Evaluating Predictive Uncertainty Visual Object Classification, and Recognizing Textual Entail- ment, MLCW'OS 05. pp. 177-190, Berlin, Heidelberg, 2006. Springer- ISBN 3-540-33427- 0. 978-3-540-33427-9, doi: 10.1007/11136679099 URL http: /xx doi org
11736790 9
Jacob Devlin, Ming- Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint 1 2018.
Xiaohan Ding, Guiguang Ding, Xiangxin Zhou, Yuchen Guo, Jungong Han, and Ji Liu. Global sparse momentum SGD for pruning very deep neural networks. In Hanna M. Wallach, Hugo Larochelle, Alina Beygelzimer, Florence d' Alché-Bue, Emily B. Fox, and Roman Garnett (eds.), Advances in Neural Information Processing Systems 32: Annual Conference on Neural Informa- tion Processing Systems 2019, NeurlPS 2019, December 8- 8-14, 2019, Vancouver, BC, Canada, pp. 6379 6091, 2019.
William Dolan and Chris Brockett. Automatically constructing a corpus of sentential paraphrases. In Proceedings of the Third International Workshop Paraphrasing (IWP2005), 2005.
Danilo Giampiccolo, Bernardo Magnini, Ido Dagan, and Bill Dolan. The third PASCAL recognizing textual entailment challenge. In Proceedings of the ACL-PASCAL Workshop Textual Entail- ment and Paraphrasing, pp. 1-9, Prague, June 2007. Association for Computational Linguistics. URL httpt:::p//.o.. www. C web. http:///aooooooooyyoooooo
Jia Guo, Minghao Chen, Yao Hu, Chen Zhu, Xiaofei He, and Deng Cai. Reducing the teacher- student gap via spherical knowledge disitllation. arXiv preprint arXiv:2010007485, 2020.
Song Han, Huizi Mao, and William J Dally. Deep compression: Compressing deep neural networks with pruning, trained quantization and huffman coding. arXiv preprint arXiv: 15.0001499 2015a.
Song Han, Jeff Pool, John Tran, and William J Dally. Learning both weights and connections for efficient neural networks. arXiv preprint arXiv: 1506.02666 2015b.
10

---

Published as a conference paper at ICLR 2023
Pengcheng He, Jianfeng Gao, and Weizhu Chen. Debertav3: Improving deberta using electra-style pre-training with 2021.
embedding sharing. arXiv preprint arXiv:2111.09543,
Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. Distilling the knowledge in neural network. arXiv preprint arXiv::503. 02531, 2015.
Lu Hou, Zhiqi Huang, Lifeng Shang, Xin Jiang, Xiao Chen, and Qun Liu. Dynabert: Dynamic bert with adaptive width and depth. arXiv preprint arXiv:2004. 040377 2020.
Xiaoqi Jiao, Yichun Yin, Lifeng Shang, Xin Jiang, Xiao Chen, Linlin Li, Fang Wang, and Qun Liu. Tinybert: Distilling bert for natural language understanding. arXiv preprint urXiv:/9099 1909.1035, 2019.
Xiao Jin, Baoyun Peng, Yichao Wu, Yu Liu, Jiaheng Liu, Ding Liang, Junjie Yan, and Xiaolin Hu. Knowledge distillation via route constrained optimization. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 1345-1354, 2019.
Simran Khanuja, Melvin Johnson, and Partha Talukdar. Mergedistill: Merging pre-trained language models using distillation. arXiv preprint arXiv:2106.02834 2021.
Young Jin Kim and Hany Hassan Awadalla. Fastformers: Highly efficient transformer models for natural language understanding. arXiv preprint arXiv:2010. 13382, 2020.
Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980 2014.
François Lagunas, Ella Charlaix, Victor Sanh, and Alexander M Rush. Block pruning for faster transformers. arXiv preprint arXiv:2109.04836, 2021.
Yann LeCun, John S Denker, and Sara A Solla. Optimal brain damage. In Advances in neural information processing systems, pp. 598 605, 1990.
Namhoon Lee, Thalaiyasingam Ajanthan, and Philip H. S. Torr. Snip: single-shot network pruning based on connection sensitivity. In 7th International Conference on Learning Representations, ICLR 2019, New Orleans, LA, USA, May 6-9, 2019. OpenReview.net, 2019.
Lei Li, Yankai Lin, Shuhuai Ren, Peng Li, Jie Zhou, and Xu Sun. Dynamic knowledge distillation for pre-trained language models. arXiv preprint arXiv:2109.11220 2021.
Chen Liang, Simiao Zuo, Minshuo Chen, Haoming Jiang, Xiaodong Liu, Pengcheng He, Tuo Zhao, and Weizhu Chen. Super tickets in pre-trained language models: From model compression to improving generalization. arXiv preprint arXiv:2105.122 12002, 2021.
Kevin J Liang, Weituo Hao, Dinghan Shen, Yufan Zhou, Weizhu Chen, Changyou Chen, and Lawrence Carin. Mixkd: Towards efficient distillation of large-scale language models. arXiv preprint arXiv:2011.00599 2020.
Ekdeep Singh Lubana and Robert P Dick. A gradient flow framework for analyzing network pruning. arXiv preprint arXiv:2009. 11899, 2020.
Paul Michel, Omer Levy, and Graham Neubig. Are sixteen heads really better than one? arXiv preprint arXiv:1905. 10650, 2019.
Seyed Iman Mirzadeh, Mehrdad Farajtabar, Ang Li, Nir Levine, Akihiro Matsukawa, and Hassan Ghasemzadeh. Improved knowledge distillation via teacher assistant. In Proceedings the AAAI Conference Artificial Intelligence, volume 34, pp. 5191-5198, 2020.
Pavlo Molchanov, Stephen Tyree, Tero Karras, Timo Aila, and Jan Kautz. Pruning convolutional neural networks for resource efficient inference. In 5th International Conference Learning Representations, ICLR 2017, Toulon, France, April 24-26, 2017, Conference Track Proceedings. OpenReview.net, 2017.
11

---

Published as a conference paper at ICLR 2023
Pavlo Molchanov, Arun Mallya, Stephen Tyree, luri Frosio, and Jan Kautz. Importance estimation for neural network pruning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 11264-11272, 2019.
Michael C Mozer and Paul Smolensky. Skeletonization: A technique for trimming the fat from a network via relevance assessment. In Advances in neural information processing systems, pp- 107-115, 1989.
Michela Paganini and Jessica Forde. On iterative neural network pruning, reinitialization, and the similarity of masks. arXiv preprint arXiv:2001.05054 2020.
Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al. Language models are unsupervised multitask learners. OpenA/ blog, 1(8):9, 2019.
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. Exploring the limits of transfer learning with a unified text-to-text transformer. arXiv preprint arXiv: 1910. 10683, 2019.
Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. Squad: 100,000+ questions for machine comprehension of text. arXiv preprint (XXv:1606.002500 2016a.
Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. SQuAD: 100,000+ questions for machine comprehension of text. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pp. 2383-2393 Austin, Texas, November 2016b. Association
for Computational Linguistics. doi: 10.18653/v1/D16 1264. URL https: www. aclweb.
atgg/anthclogy/D... 6 1264
Pranav Rajpurkar, Robin Jía, and Percy Liang. Know what you don't 't know: Unanswerable questions 2018.
for squad. arXiv preprint 1806.
Alex Renda, Jonathan Frankle, and Michael Carbin. Comparing rewinding and fine-tuning in neural network pruning. In 8th International Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020. OpenReview.net, 2020.
Mehdi Rezagholizadeh, Aref Jafari, Puneeth Salad, Pranav Sharma, Ali Saheb Pasand, and Ali Ghodsi. Pro-kd: Progressive distillation by following the footsteps of the teacher. arXiv preprint arXiv:2110.08532 2021.
Adriana Romero, Nicolas Ballas, Samira Ebrahimi Kahou, Antoine Chassang, Carlo Gatta, and Yoshua Bengio. Fitnets: Hints for thin deep nets. arXiv preprint arXiv:1412.666 2014.
Victor Sanh, Lysandre Debut, Julien Chaumond, and Thomas Wolf. Distilbert, a distilled version of bert: smaller, faster, cheaper and lighter. arXiv preprint arXiv:1910.07 2019.
Victor Sanh, Thomas Wolf, and Alexander Rush. Movement pruning: Adaptive sparsity by fine- tuning, arXiv preprint 2XXi:20005.07683 2020.
Wenxian Shi, Yuxuan Song, Hao Zhou, Bohan Li, and Lei Li. Follow your path: a progressive method for knowledge distillation. In Joint European Conference on Machine Learning and Knowledge Discovery in Databases, pp. 596-611. Springer, 2021.
Richard Socher, Alex Perelygin, Jean Wu, Jason Chuang, Christopher D Manning, Andrew Ng. and Christopher Potts. Recursive deep models for semantic compositionality over a sentiment treebank. In Proceedings of the 2013 conference empirical methods in natural language pro- cessing, pp. 1631-1642, 2013.
Siqi Sun, Yu Cheng, Zhe Gan, and Jingjing Liu. Patient knowledge distillation for bert model compression. arXiv preprint
2019.
Zhiqing Sun, Hongkun Yu, Xiaodan Song, Renjie Liu, Yiming Yang, and Denny Zhou. Mobile- bert: a compact task-agnostic bert for resource- resource-limited devices. arXiv preprint arXiv:2004.02984 2020.
12

---

Published as a conference paper at ICLR 2023
Raphael Tang, Yao Lu, Linqing Liu, Lili Mou, Olga Vechtomova, and Jimmy Lin. Distilling task- specific knowledge from bert into simple neural networks. arXiv preprint 2019.
Lucas Theis, Iryna Korshunova, Alykhan Tejani, and Ferenc Huszár. Faster gaze prediction with dense networks and fisher pruning. arXiv preprint arXiv: 18Xx:18011.05777, 2018.
Iulia Ture, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. Well-read students learn better: On the importance of pre-training compact models. arXiv preprint arXiv: /908. 08962, 2019.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention all you need. Advances in neural informa- fion processing systems, 30, 2017.
Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R Bowman. Glue: A multi-task benchmark and analysis platform for natural language understanding, arXiv preprint 1804 2018.
Wenhui Wang, Hangbo Bao, Shaohan Huang, Li Dong, and Furu Wei. Minilmv2: Multi- head self-attention relation distillation for compressing pretrained transformers. arXiv preprint arXiv:2012.15828, 2020a.
Wenhui Wang, Furu Wei, Li Dong, Hangbo Bao, Nan Yang, and Ming Zhou. Minilm: Deep self- attention distillation for task-agnostic compression of pre-trained transformers. arXiv preprint
arXiv:2002.10997, 2020..
Ziheng Wang, Jeremy Wohlwend, and Tao Lei. Structured pruning of large language models. arXiv preprint arXiv:1910.047:5 2019.
Alex Warstadt, Amanpreet Singh, and Samuel R Bowman. Neural network acceptability judgments. Transactions of the Association for Computational Linguistics, 7:625-641, 2019..
Adina Williams, Nikita Nangia, and Samuel Bowman. A broad-coverage challenge corpus for sen- tence understanding through inference. In Proceedings of the 2018 Conference of the North Amer- ican Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers), pp. 1112-111 122. Association for Computational Linguistics, 2018. URL http: //t:://ac../ web. org antho loghoooogyy///
11C1
Mengzhou Xia, Zexuan Zhong, and Danqi Chen. Structured pruning learns compact and accurate models. arXiv preprint wrXi::2044.00408 2022.
Xia Xiao, Zigeng Wang, and Sanguthevar Rajasekaran. Autoprune: Automatic network pruning by regularizing auxiliary parameters. Advances in neural information processing systems, 32, 2019.
Canwen Xu, Wangchunshu Zhou, Tao Ge, Furu Wei, and Ming Zhou. of Compressing bert by progressive module replacing. arXiv preprint arXiv:2002.02925. 2020.
Dongkuan Xu, Ian EH Yen, Jinxi Zhao, and Zhibin Xiao. Rethinking network pruning-under the pre-train and fine-tune paradigm. arXiv preprint arXiv:2104.08682 2021.
Ofir Zafrir, Ariel Larey, Guy Boudoukh, Haihao Shen, and Moshe Wasserblat. Prune once for all: Sparse pre-trained language models. Advances in Neural Information Processing Systems, 2021.
Qingru Zhang, Simiao Zuo, Chen Liang, Alexander Bukharin, Pengcheng He, Weizhu Chen, and Tuo Zhao. Platon: Pruning large transformer models with upper confidence bound of weight importance. In International Conference Machine Learning, pp. 26809-26823. PMLR, 2022.
Wangchunshu Zhou, Canwen Xu, and Julian McAuley. Bert learns teach: Knowledge distillation with meta learning. In Proceedings of the 60th Annual Meeting of the Association for Computa- tional Linguistics (Volume 1: Long Papers), pp. 7037-7049, 2022.
Michael Zhu and Suyog Gupta. To prune, or not to prune: exploring the efficacy of pruning for model compression. arXiv preprint arXiv:1710.011 2017.
13

---

Published as a conference paper at ICLR 2023
Yukun Zhu, Ryan Kiros, Rich Zemel, Ruslan Salakhutdinov, Raquel Urtasun, Antonio Torralba, and Sanja Fidler. Aligning books and movies: Towards story-like visual explanations by watching movies and reading books. In The IEEE International Conference on Computer Vision (ICCV), December 2015.
14

---

Published as a conference paper at ICLR 2023
A APPENDIX
A.1 DATA
Continual Pre-training. We use the same pre- pre-training data as BERT: Wikipedia (English Wikipedia dump8; 12GB) and BookCorpus ((Zhu et al. 2015)) (6GB). We clean the corpus by removing ta- bles, lists and references following BERT. We then pre-process the cleaned corpus by concatenating all sentences in paragraph and truncating the concatenated passage by length of 128 following We tokenize the corpus with the vocabulary of BERT (30k).
TinyBERT JJio et al. 2
Fine-tuning. GLUE is commonly used natural language understanding benchmark containing nine tasks. The benchmark includes question answering (Rajpurkar et al. 2016bb. linguistic accept- ability (CoLA. Warstadt et al. 2019), sentiment analysis (SST, /Socher al. 2013), text similarity (STS-B, Cer et al. 2017), paraphrase detection (MRPC. Dolan & Brockett 2003). and natural lan- guage inference (RTE& & MNLL,Daggan
al. [2066] Bar-Haim al. [2006] Giampiccolo et al. 2007] Bentivogli et al. 2009: Williams et al. 2018) tasks. Details of the GLUE benchmark, including tasks, statisties, and evaluation metrics, are summarized in Table T6] SQuAD v1.1/v2.0 are the Stan- ford Question Answering Datasets (Rajpurkar et al.) [018 2016a), two popular machine reading comprehension benchmarks from approximately 500 Wikipedia articles with questions and answers obtained by crowdsourcing. The SQuAD v2.0 dataset includes unanswerable questions about the same paragraphs.
A.2 CONTINUAL PRE-TNAINING IMPLEMENTATIONS
configurations for continual pre-training HomoBERT models on the open-domain data. We set the distillation temperature as 2. We empirically observe setting ty within the range of 0.5T ~ 0.97 can achieve similarly good downstream performances. Fur- thermore, observe that different weight modules may prefer different tys: 1) it is better to finish the pruning of output projection matrices in the attention module, the feed-forward module and the embedding module early, because pruning them late will induce large increment in distillation loss and the student performance is difficult to recover. 2) the student performance is less sensitive to the pruning of key and query projection matrices in the attention module and the input projection matrix in the feed-forward module, and can often casily recover. Based on this observation, we set f 0.5 for the output projection matrices in the attention module, the feed-forward module and the embedding module. For key and query projection matrices in the attention module and the input projection matrix in the feed-forward module, we set t, 0.9. For other matrices, we set ty = 0.7. This configuration brings around small and consistent gain of 0.05 ~ 0.08 on GLUE. The continual pre-training experiment runs for around 13 hours on Nvidia A100 GPUs.
Table Table@@preseen presents the
Table 6: Hyper-parameter configurations for task task-agnostic distillation of HomoBERT models.
Hypeerparammttrs
HomoBERT-base
HomoBERT-s
HomoBERT-xs
HomoBERT-iiny
Leariing Rates
x 10-4
6 10
6 x 10-4
6 10
Batch Size
4000
Training Epochs
Learning Rate Decay
Linear
Learning Rate Warmup
0.1
Max Sequence Length
128
Weight Decay
0.01
Adam &
0.9
Adam &2
0.999
Adam e
10-6
Gradient Clipping
None
01 022 ag. Ot4
f
hwwpy//wwwnnnnotooimooooeooeeonn training data.py
15

---

Published as a conference paper at ICLR 2023
A.3 FINE-TUNING IMPLEMENTATIONS
Table 7 presents the hyper-parameter configurations for fine-tuning HomoBERT models on the GLUE benchmark. We fine-tune the MRPC, RTE and STS-B from fine-tuned MNLI student. All experiments are conducted on Nvidia A100 GPU.
Table 7: Hyper-parameter configurations for fine-tuning HomoBERT models on the GLUE bench- mark. "Epoch" refers to the total training epochs; we adopt early-stopping strategy in practice.
Hyper-parameters
HomoBERT-buse
HomoBERT-s
HomoBERT-xs
HomoBERT-tiny
Learning Rates
(2,3,4.5] x 10 (4,5,6, 7 10 5
{6. 7)
(6. 7)
Batch Size
16 for RTE and MRPC: 32 for the others.
Training Epochs
3 for MNLI and QNLI; 6 for the others.
Learning Rate Decay
Linear
Learning Rate Warmup
Max Sequence Length
128
Dropout of Task Layer
0.1
Weight Decay
Adam 51
Adam D2
0.999
Adam
10-6
Gradient Clipping
Table 8 presents the hyper-parameter configurations for fine-tuning HomoBERT models on the SQuAD v1.1/2.0. All experiments are conducted on 1 Nvidia A100 GPU.
Table 8: Hyper-parameter configurations for fine-tuning HomoBERT models SQuAD v1.
Hyper-parameters
HomoBERT-s<sshin
Learning Rates
x 10
Batch Size
12
Training Epochs
2
Leurning Rate Decay
Lineur
Leurning Rate Warmup
0.2
Max Sequence Length
384
Dropout of Task Layer
Weight Decay
0
Adam 8
0.9
Adam &g
0.999
Adam
10-6
Gradient Clipping
A.4 COMPARISON WITH
DISTILLATION METHODS
Mable@Ccompace HosoDistil with commonly used task-specific distillation baseline methods: PKD (Sun al. 2019), BERT-of- of-TT-o--Thesuus Xu al. 2020), MixKD (Liang et al. 2020). DynaBERT (Hou et al. 2020). ProKT (Shi et al.) (021) and MetaDistil (Zhou et al.) 2022). All baseline methods use BERT-base fine-tuned the target task the teacher model, and 6-layer pre-trained BERT-base as the initialization of the student model. The student model is then distilled with the target task data. As shown in the Table 9] HomoDistil demonstrates a prominent margin over the commonly used task-specific methods.
A.5 A COMPARISON WITH "PRUNING+DISTILLLATION METHODS
We elaborate our discussion Section 6 by comparing HomoDistil and the existing methods that combining pruning and distillation in more details. Table I0 and Table II present the detailed
comparison among Homtlistil, DynaBERT (Hou et al. 2020). SparseBERT (Xu et al. (021) and
CoFi Xia et al. 2022). We also list the major differences below:
HomoDistil focuses on the task-agnostic setting. In the task-specific setting, pruning incurs an inevitable loss of task-relevan, pre-training knowledge that may not be present in the fine-tuning data. Therefore, existing works leave the word embeddings untouched (e.g., take up around 20 million parameters in BERT-base). In contrast, this problem does not exist in the task-agnostic
16

---

Published as a conference paper at ICLR 2023
Table 9: The evaluation performance of HomoDistil and the commonly used task-specific distillation baseline methods on the GLUE development set. The results of PKD (Sun et al., 2019) and ProKT
(Shi et al. 2021) are from our implementation. The rest results are from the original papers.
Model
Params
MNLI m/mm
QQP
QNLI
SST-2
CoLA Acc
RTE
MRPC
STS-B
Avg
(million)
Acc
Acc
Acc
Acc
Acc
Acc
Spearman
Score
PKDs Sun al. 2019
66
81.3/-
88.4
88.4
91.3
45.5
66.5
85.7
86.2
79.2
BERT of Theseusg [NB et al.
2020
66
82.3/-
89.5
89.5
91.5
51.1
68.2
89.0
88.7
81.2
MixKD; (Liung
2020
66
82.5/-
90.8
88.8
92.1
67.9
84.1
DynaBEKI, [Ho0 Wed. 2020
83,7/84.6
91.1
90.6
92.7
54.6
66.1
85.0
88.6
81.6
ProKT, (Shi di 20M4
66
82.8/83.2
90.9
89.7
91.3
54.3
68.4
86.3
88.9
81.6
MetaDistile [Zhou of 2022
83,5/83.8
91.0
90.4
92.3
58.6
69.4
86.8
89.1
82.6
HomoBERT-base
65
84,2/84.3
91.2
90,7
92.7
55,9
77.6
89.0
89.2
83,8
Table 10: Comparison of HomoDistil and the existing "Pruning-Distillatoon methods from the distillation perspective.
Method
Distillation Setting
Teacher Model
Student Initialization
DynaBERT (How et al.
2020
Task specifii
Fine tuned weights
Pruned, pre trained weights Pre-trained weights Pre trained weights
CoFi (Xia et aL 2022
Task-specific
Fine-tuned weights
SparseBERK I XX at. 2021
Task specific
Fine tuned weights
HomoDistil
Tusk-agnostic
Pre-trained weights
Pre-trained weights
Table 11: A comparison of HomoDistil and the existing Brrniing-Dstillannn methods from the
pruning perspective.
Method
Pruning Setting
Pruning Criterion
Pruning Granularity
Controllable Layer Width
DynaBERT
First prune then distill
Head and FFN sensitivity
Head, FFN Layer, head, FPN, weight Weight
No
Prune while distill
to regularization
SparseBERT
Prune while distill
Weight magnitude
No
HomaDistil
Prune while distill
Column sensitivity
Row, column
Yes
setting. This allows prune the word embeddings and produce smaller model more suitable for edge devices (e.g., around 15 million parameters). Furthermore, a task-specific model needs to be specifically pruned for each individual task, while a task-agnostic model can be fine-tuned for any task with a low cost.
HomoDistil initializes the student with the teacher. To maintain small discrepancy in the early stage, HomoDistil initializes the student with the teacher. In contrast, DynaBERT initializes the student with a target-size subnetwork. SER and CoFi initialize the student with pre-trained weights while the teacher with fine-tuned weights.
HomoDisti simultancously prunes and distills and allows interactions between them. To main- tain small discrepancy throughout distillation, HomoDistil prunes based on the sensitivity to make the pruning operation "distillation-wwar aware"". Specifically, HomoDistil selects the columns and rows to prune based on their contributions to the distillation loss. In contrast, DynaBERT treats pruning and distillation as two independent operations by first pruning then distilling the subnetwork. Sparse- BERT prunes based on the weight magnitude without considering the influence on the distillation loss.
HomoDistil prunes rows and columns. The granularity of rows and columns is sufficiently small to control the increment in discrepancy while maintaining the practical benefits of structured pruning.
HomoDistil can control the layer width. HomoDistil enforees local sparsity constraint for each matrix, producing a model with consistent width in each layer. In contrast, SparseBERT and CoFi have control over the layer width, which might result in wide matrices as the memory bottlenecks.
Table 12 shows the evaluation performance of HomoDistil, CoFi and SparseBERT on the GLUE benchmark (DynaBERT results are presented in Table 9 We see that HomoDistil achieves noticeable gain over CoFi and a comparable performance with SparseBERT with nearly half their sizes.
17

---

Published as a conference paper at ICLR 2023
Table 12: The performance comparison with the existing Mraniing-Dstiilaaninn methods. All
sults are reported by their papers.
Model
Params (million)
mNLImm/mm
QQP
QNLI
SST-2
CoL.A
RTE
MRPC
STS-B Spearman
Avg Score
Acc
Acc
Acc
Acc
Acc
Acc
Cobles Xia et all 2022
28.4
80.6/-
90.1
86.1
90.6
35.6
64.7
82.6
83.1
76.7
SpasseBEK1gg Xu
88.5
HomoBERT-xsmall
15.6
81,5/81.
90.0
88.0
90.3
40.8
71.5
87.7
88.0
79.7
A.6 COMPUTATIONAL COSTS
Table 13 compares the computational costs of HomoDistil and the baseline methods during infer- ence. We profile the inference time and the number of FLOPs (embedding excluded) during the forward pass using the profiler package released by pytorch [ We conduct the measurements on the GLUE development set with a batch size of 128 and maximum sequence length of 128 on one Nvidia A100 GPU. We compute the averaged time and FLOPs over all batches. The speedup computed with respect to BERT-base. For a fair comparison, we only compare with compact models.
As can be observed, HomoDistil achieves some inference speedup and FLOPs reduction, but not as much the other models under a similar parameter budget. This is because HomoDistil allocates higher budget to the backbone parameters and a lower budget to the embedding parameters. How- remark that HomoDistil achieves better accuracy and enjoys the same (or more) storage benefits than the distilled (or structured pruned) models.
Table 13:
The inference speedup and the number of FLOPs (embedding excluded) of HomoDistil and the baseline methods. The speedup is computed with respect to BERT-base.
Model
Params (million)
Inference Spesdup
FLOPs (non-embeddiing)
BERT base
109
1.00x
1.00x
DisiiBERT, TnyBBERIe-GGD
0.50x
66
1.98x
0,50x
MiniLMvla,
0.50x
Minil Mv2,
66
1.98
0.50
HomoBERT base
1.30x
0.56x
BERT-small
4.77
0.15x
CoFis%
28.4
5.16x
0.05x
TinyBERTasass
MiniLMvla
17.0
7.34x
0.06x
TinyBERT
14.5
HomoBERTTsmall
17.1
2,40x
0.11x
HomoBERT xsmall
15.6
2.51x
0.10x
HomoBERT-inn
14.1
2.55x
0.09x
A.7 STATISTICS OF EXPERIMENTAL RESULTS
All experimental results of HomoDistil presented in this paper the median of five random seeds. Table 14 and Table 15 show the standard deviations of the experimental results on the GLUE bench- mark (Table 2) and on the SQuAD v1.1/2.0 datasets (Table respectively.
Table 14: The standard deviation of the experimental results GLUE development set in Table
MNLL-m/mm Acc
QQP
QNLI
SST-2
CoLA Acc
RTE
MRPC Acc
STS-B P/S
Model
Acc/F1
Acc
Acc
Ace
HomoBERRThhae
0.13/0.20
0.09/0.12
0.34
0.24
1.72
0.93
1,770.83
0.15/0.18
HomoBERT-small
0.23/0.14
0.08/0.20 20
0.14
0.27
2.49
1.29
0.25/1.43
0.23/0.25
HomoBERT-xxmm
14/0//112
0.08/0.10
0.24
0.61
1.32
1.29
0.95/0.66
0.27/0.28
HomoBERT--iny
0610/0
0.11/0.13
0.29
0.16
1.26
1.26
1.05/0.62
0.19/0.22
httt:////ggyocaaaanoaaaaaaieeeer recipe.html
18

---

Published as a conference paper at ICLR 2023
Table 15: The standard deviation of the experimental results on SQuAD v1.1/2.0 in Table 3
Model
SQuAD SQuAD SQuAD SQuAD
HomoBERT
0.24
0.19
0.37
0.37
HomoBERT-xsmall
0.15
0.19
0.62
0.59
HomoBERT-uny
0.29
0.31
0.72
Table 16: Summary of the GLUE benchmark.
Corpus
Task
#Train
#Dev
#Test
#Label
Metries
Single-Sentence Clussification (GLUE)
CoLA
Acceptability
8.5k
1k
1k
2
Matthews corr
SST
Sentiment
67k
872
1.8k
2
Accuracy
Pairwise Text Classification (GLUE)
MNLI
NLJ
393k
20k
20k
3
Accuracy
RTE
NLI
2.5k
276
3k
2
Accuracy
QQP
Paraphrase
364k
40k
391k
2
Accuracy/Fi
MRPC
Paraphrase
3.7k
408
1.7k
2
Accuracy/FI
QNLI
QA/LLI
108k
5.7k
5.7k
2
Accuracy
Text Similarity (GLUE)
STS
Similarity
1.5k
1.4k
Pearson/Spearman carr
19


Reviews:
[{"review_id": "nhd1ymtNW-", "reviewer": "Reviewer_1n6p", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The authors propose a task-agnostic knowledge distillation method for NLP models based on iterative pruning combined with distillation. A novel distillation loss is proposed, that performs differentiable matching of the latent representations. The proposed distillation method is compared to a selection of baselines, and is shown to perform well.", "summary_of_the_review": "Overall, I think this is a useful, practical paper, showing how existing methods and ideas can be combined to achieve higher quality solutions. Compressing huge models is an important task, since many resource-constrained settings exist that currently can’t leverage the modern NLP models. The authors should clarify if the latent representation alignment via a differentiable weight matrix is their idea: this is obviously an excellent way of discovering the alignment, I have not encountered it in the literature before. However, if the idea is borrowed, the sources must be stated clearly.\n\nSee below for additional suggested corrections:\n\nPage 4: “the loss for continual pre-training the student model on the open-domain data” -> the loss for continual pre-training of the student model on the open-domain data\nPage 4: “consists both the distillation…” -> consists of both…\n\nMedians of 5 runs are reported - what about the corresponding standard deviations? Perhaps Table 2 can be represented visually (e.g. box-and-whisker plots)?\n\nRegarding the matching rule: you propose a differentiable weighted matching scheme, i.e. the matching is determined via training. Is my understanding correct? Has similar matching been applied in the NLP domain before? Please indicate the sources.", "strength_and_weaknesses": "Strengths:\n* An efficient distillation method is proposed that leverages both KD and pruning\n* Experiments are realistic, and show the superiority of the proposed method compared to the SOTA baselines\n* Latent space mapping/alignment is done elegantly via a learned weight matrix\n\nWeaknesses:\n* Reported medians over 5 runs, no standard deviations: how robust is the method?\n* Table 2 could be presented visually", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "TPuaa__W3bl", "reviewer": "Reviewer_Y2q6", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper propose HomoDistil, a method which combines knowledge distillation and pruning to obtain efficient networks.", "summary_of_the_review": "Although the proposed method is simple and seems effective, I lean towards rejection because of the limited novelty.", "strength_and_weaknesses": "The proposed method is simple yet effective, good performance are obtained according to experiments;\n\nThe paper is easy to follow;\n\nHowever, the proposed method has limited novelty, it is based on some existing methods;\n\nIt would be nice if the authors can provide a comprehensive comparison of computation cost/time between the proposed method and previous methods;\n\nWhy results of model with 66M parameters is not reported in Table 3?\n\nPlease adjust the position of Table 2, Table 3 and the paragraph between the two tables.\n\nSome related works are suggested to be compared.\n\n[1]. Multi-Granularity Structural Knowledge Distillation for Language Model Compression. Chang Liu, Chongyang Tao, Jiazhan Feng, Dongyan Zhao\n\n[2]. BERT Learns to Teach: Knowledge Distillation with Meta Learning. Wangchunshu Zhou, Canwen Xu, Julian McAuley.\n\n[3]. LRC-BERT: Latent-representation Contrastive Knowledge Distillation for Natural Language Understanding. Hao Fu, Shaojun Zhou, Qihong Yang, Junjie Tang, Guiquan Liu, Kaikui Liu, Xiaolong Li.\n\n[4]. MixKD: Towards Efficient Distillation of Large-scale Language Models. Kevin J Liang, Weituo Hao, Dinghan Shen, Yufan Zhou, Weizhu Chen, Changyou Chen, Lawrence Carin.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "9amM0PJPZIs", "reviewer": "Reviewer_na3M", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper tackles with a problem that when distilling a small student model using a large pre-trained transformer, the “capacity gap” between the student and the teacher often hinders the teacher from transferring good performance to the student. The HomoDistill method proposed by this paper investigates to progressively prune the teacher model into a small student model, while in the same time distilling it using the original teacher model. Experiments show that the combination of pruning and distillation shows advantage on several NLU and QA tasks.", "summary_of_the_review": "In general, I think HomoDistill is a novel method that combines pruning with distillation in a task-agnostic setting for NLP tasks. The experiments on NLU and QA benchmarks show the advantages of HomoDistill over other distillation methods. However, the paper lacks experimental comparison with other methods that combine pruning and distillation. Appending such comparison would make this paper more solid.", "strength_and_weaknesses": "strength:\n* HomoDistill illustrates advantageous performance over other distillation methods like TinyBERT.\n* The combination of pruning and distillation in task-agnostic setting seems novel. \n\nweakness:\n* A comparison with other prune & distill methods such as (Xu et al., 2021; Xia et al., 2022) would make this submission more complete.\n* Please also discuss the difference between HomoDistill and [1], which presents a similar method for vision tasks, where a teacher model is slimmed into a small student model while distilling it.\n\n[1] Slimmable Neueral Networks, ICLR 2019.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Decision: Accept: poster