AutoSKDBERT: Learn to Stochastically Distill BERT
Authors: Zixiang Ding
GUO-QING JIANG
Shuai Zhang
Lin Guo
Wei Lin
Venue: Submitted to ICLR 2023 (2023)
Keywords: Categorical distribution optimization
Stochastic knowledge distillation
BERT compression
GLUE

Abstract: In this paper, we propose AutoSKDBERT, a new knowledge distillation paradigm for BERT compression, that stochastically samples a teacher from a predefined teacher team following a categorical distribution in each step, to transfer knowledge into student. AutoSKDBERT aims to discover the optimal categorical distribution which plays an important role to achieve high performance. The optimization procedure of AutoSKDBERT can be divided into two phases: 1) phase-1 optimization distinguishes effective teachers from ineffective teachers, and 2) phase-2 optimization further optimizes the sampling weights of the effective teachers to obtain satisfactory categorical distribution. Moreover, after phase-1 optimization completion, AutoSKDBERT adopts teacher selection strategy to discard the ineffective teachers whose sampling weights are assigned to the effective teachers. Particularly, to alleviate the gap between categorical distribution optimization and evaluation, we also propose a stochastic single-weight optimization strategy which only updates the weight of the sampled teacher in each step. Extensive experiments on GLUE benchmark show that the proposed AutoSKDBERT achieves state-of-the-art score compared to previous compression approaches on several downstream tasks, including pushing MRPC F1 and accuracy to 93.2 (0.6 point absolute improvement) and 90.7 (1.2 point absolute improvement), RTE accuracy to 76.9 (2.9 point absolute improvement).

Under review as a conference paper at ICLR 2023
AUTOSKDBERT: LEARN TO STOCHASTICALLY DIS- TILL BERT
Anonymous authors Paper under double-blind review
ABSTRACT
In this paper, we propose AutoSKDBERT, new knowledge distillation paradigm for BERT compression, that stochastically samples a teacher from a predefined teacher team following categorical distribution in each step, to transfer knowl- edge into student. AutoSKDBERT aims to discover the optimal categorical distri- bution which plays an important role achieve high performance. The optimiza- tion procedure of AutoSKDBERT can be divided into two phases: 1) phase-1 op- timization distinguishes effective teachers from ineffective teachers, and 2) phase- 2 optimization further optimizes the sampling weights of the effective teachers to obtain satisfactory categorical distribution. Moreover, after phase- optimiza- tion completion, AutoSKDBERT adopts teacher selection strategy to discard the ineffective teachers whose sampling weights are assigned to the effective teach- ers. Particularly, to alleviate the gap between categorical distribution optimization and evaluation, we also propose stochastic single-weight optimization strategy which only updates the weight of the sampled teacher each step. Extensive ex- periments on GLUE benchmark show that the proposed AutoSKDBERT achieves state-of-the-arr score compared previous compression approaches on several downstream tasks, including pushing MRPC F1 and accuracy to 93.2 (0.6 point absolute improvement) and 90.7 (1.2 point absolute improvement), RTE accuracy to 76.9 (2.9 point absolute improvement).
INTRODUCTION
BERT (Devlin et al., 2019) has brought about a sea change in the field of Natural Language Process- ing (NLP). Following BERT, numerous subsequent works focus various perspectives to further improve its performance, e.g., hyper-parameter (Liu et al., 2019b), pre-training corpus (Liu al., 2019b; Raffel et al., 2020), learnable embedding paradigm (Raffel et al., 2020), pre-training task (Clark et al., 2020), architecture (Gao et al., 2022) and self-attention (Shi et al., 2021), etc. How- ever, there massive redundancies in the above BERT-style models w.r.t. attention heads (Michel et al., 2019; Dong et al., 2021), weights (Gordon et al., 2020), and layers (Fan et al., 2020). Conse- quently, many compact BERT-style language models proposed via pruning (Fan al., 2020; Guo et al., 2019), quantization (Shen et al., 2020), parameter sharing (Lan et al., 2020) and Knowledge Distillation (KD) (landola al., 2020; Pan al., 2021). In this paper, we focus on the KD-based compression approaches.
From the point of view of learning procedure, KD is used in the pre-training (Ture al., 2019; Sanh et al., 2019; Sun et al., 2020; Jiao et al., 2020) and fine-tuning phases (Sun et al., 2019; Jiao et al., 2020: Wu al., 2021). On the other hand, from the point of view of distillation objective, KD is employed for the outputs of hidden layer (Sun et al., 2020), final layer (Wu et al., 2021). embedding (Sanh et al., 2019) and self-attention (Wang al., 2020). Wu et al. (2021) employ multiple teachers to achieve better performance than single-teacher KD based approaches on several downstream tasks of GLUE benchmark (Wang et al., 2019). As shown in Table 1, nevertheless, the ensemble of multiple teachers are not always more effective than the single teacher for student distillation. There are two possible reasons: 1) diversity losing (Tran et al., 2020) and 2) capacity gap (Mirzadeh et al., 2020). On the one hand, the ensemble prediction of multi-teacher KD loses the diversity of each teacher. On the other hand, between the large-capacity teacher ensemble and small-capacity student, there is capacity gap which can be prone to unsatisfactory distillation performance.

---

Under review as a conference paper at ICLR 2023
Table 1: Performances of knowledge distillation using single and multiple teachers for 6-layer BERT-style language model on the development set of GLUE benchmark. this experiment. we employ five teachers, i.e T1o to T14 shown in Appendix C.I. for single-teacher distillation and multi-teacher distillation. We introduce the implementation details in Appendix H.
Task
MRPC
RTE
CoLA
SST-2
QQP FTiace
QNLI
MNLI
Metrics
PT
Mec
Best Single Teacheri
T13
T10
T10
T12
T11
T12
T143
Single-teacher KD
90.0
73.3
49.3
93.1
89.0
91.4
83.5
Multi-teacher KD)
89.7
73.7
50.1
92.2
88.6
91.1
83.6
Gain
-0.3
-0,9
-0.4
The teacher for student distillation each downstream task shown Table Pre-training with whole word masking.
To solve the above mentioned issues, we propose AutoSKDBERT which stochastically samples teacher from predefined teacher team following a categorical distribution in each step, to transfer knowledge into student. The task of AutoSKDBERT is learning the optimal categorical distribu- tion to achieve high performance. 1) Given a teacher team which consists of multiple teachers with multi-level capacities, AutOSKDBERT optimizes initialized categorical distribution distinguish effective teachers from ineffective teachers in phase- optimization. 2) The sampling weights of the ineffective teachers are assigned to the effective teachers via teacher selection strategy after phase-1 optimization completion. 3) AutoSKDBERT further optimizes the weights of the effective teachers rather than the ineffective teachers' in phase-2 optimization. We implement extensive experiments on GLUE benchmark (Wang et al., 2019) to verify the effectiveness of the proposed AutoSKD- BERT. Moreover, to show the generalization capacity, we have also distilled deep convolutional neural network (e.g., ResNet (He et al., 2016), Wide ResNet (Zagoruyko & Komodakis, 2016)) by AutoSKDBERT for image classification on CIFAR-I00 (Krizhevsky et al., 2009), as shown in Appendix B. Our contributions summarized as follows!:
We propose AutoSKDBERT which stochastically samples a teacher from the predefined teacher team following the categorical distribution in each step, to transfer knowledge into the student of BERT-style language model.
We propose a two-phase optimization framework with teacher selection strategy to select effective teachers and learn the optimal categorical distribution in a differentiable way.
We propose Stochastic Single-Weight Optimization (SSWO) strategy to alleviate the con- sistency gap between the categorical distribution optimization and evaluation for perfor- mance improvement.
THE PROPOSED AUTOSKDBERT
2.1 OVERVIEW
In each step, AutOSKDBERT samples a teacher I from a teacher team which consists of n multi- level BERT-style teachers Tim to transfer knowledge into student S. The objective function of AutoSKDBERT can be expressed as
((w)= = Σ
(1)
ZEX
where La represents distilled loss function to compute the difference between the student S with learnable parameter w and the sampled teacher T, X' denotes the training data, fTeTi.. (.) and fs(+)
denote the logits from T and S, respectively.
In AutoSKDBERT, categorical distribution Cat(O) where 0 {0 and D 0, 1, is em- ployed to sample the teacher from the teacher team. Particularly, the probability p(T.) of T; being sampled is 0,. We observe that (ft( plays important role for obtaining high performance of AuEOSKDBERT. As a result, the task of AutoSKDBERT then turns into learning the optimal cate- gorical distribution Cat(O*), illustrated in Figure 1.
'The code will be made publicly available upon publication of the paper.
2

---

Under review as a conference paper at ICLR 2023
Teacher Team
Student Distillation
(Sec. 2.3)
(Sec. 2.4)
Distribution
Catogorical
Student
Opsimization
Weight Freeze
Trinning
0.380
0.254
61
O2
Spli
S(w)
Training Subeet
Validation
S(wo)
Phase 2 Optimization
0.450
0.224
0.316
Mtalk
Weals
Wegght
Tencher Selection
0.055
0.075
0.400
0.195
0.275
2.4.2)
Training Data
Phase-1 Optimization
Cu
Splitt
Castpf
0.201
0.200
0.201
Training Subsect
Validation Subsei
8(w)
Categorical Distribution Initialization
Categorical
Distillation (Sec. 2.3)
Student
(Seo. 2.4) Distribation
Two-phase Optimization
Optimization
Figure I: Two-phase optimization framework with teacher selection strategy AutoSKDBERT. 1) For pre- defined teacher team, ABERTKDBERT optimizes an initialized categorical distribution to distinguish effective teachers from ineffective teachers. 2) After phase-1 optimization completion, the sampling weights of the inef- fective teachers are assigned the effective teachers via teacher selection strategy. 3) AutoSKDBERT further optimizes the weights of the effective teachers rather than the ineffective teachers phase-2 optimization. Best viewed color.
2.2 PROBLEM FORMULATION
AutOSKDBERT has two groups of learnable parameter: 1) to of student and 2) of categorical dis- tribution. We split original training data into training and validation subsets, and denote Ltroin and Leal as the losses on training and validation subsets, respectively. Both Ltrain and L are deter- mined only by Cot(O), but also by w. Particularly, AutoSKDBERT aims learn the best cate- gorical distribution Cat (8") that minimizes the validation loss Leai(w", CCa((f), where the weights w" associated with the categorical distribution Cat(O) are obtained by argmin,, Eirain(w: Cat(f)). Consequently, AutoSKDBERT can be considered as a bilevel optimization problem (Colson et al., 2007) with upper-level variable Cat(O) and lower-level variable w:
min Cas(A)
(wa (O)t(O)) Cat(O)),
(2)
" w²(Cat(f) argmin Ltrasn (w, Cat(f)).
We optimize w of student (see Section 2.3) and of categorical distribution (see Section 2.4) in an alternate and iterative way, and show the optimization algorithm in Algorithm 1.
2.3 STUDENT DISTILLATION
For student distillation, Cft( is frozen. Similar to Eq. we utilize the following object function:
[(w) calOften (4), f((;
(3)
ZEX'
where i indicates the probability of the teacher i being sampled from Tisa according to Cat(O).
3

---

Under review as a conference paper at ICLR 2023
Algorithm 1: Two-phase Optimization for AutoSKDBERT
Initialize categorical distribution Cat(O(1) for phase- optimization, weights of student,
maximum step N, current step n 0;
while < ? do
N
Update (a(() by descending Eq. 7; distribution opt Amization Update w by descending Eq. 3;
// phase-1 categorical
// phase-1 student distillation
end
Select effective teachers generate Cat( by Eq. while N
// teacher selection
7 1 n < do
Update Cat( by descending Eq. 7; distribution optim zation Update w by descending Eq. 3;
1/ phase-2 categorical
// phase-2 student distillation
1;
end
2.4 CATEGORICAL DISTRIBUTION OPTIMIZATION
For categorical distribution optimization, is frozen. We propose two-phase optimization frame- work with teacher selection strategy to learn appropriate categorical distribution:
1. Phase-1 Optimization distinguishes effective teachers from ineffective teachers in the teacher team according to Cat(A);
2. Teacher Selection discards the ineffective teachers whose weights are assigned to the fective teachers;
3. Phase-2 Optimization further optimizes the weights of the effective teachers rather than the ineffective teachers;
where a Stochastic Single-Weight Optimization (SSWO) strategy is proposed for categorical distri- bution optimization. Below, categorical distribution optimization and teacher selection strategy are introduced in detail.
2.4.1 CATRGORICAL DISTRIBUTION OPTIMIZATION VIA SSWO
To optimize Cat(O) in a differentiable way, Continuous Relaxation (CR) (Liu et al., 2019a) is common technique to obtain mixture of logits teachers
Fr.m (z; Cat(f))
Σ (x),
(4)
Subsequently, Cat be optimized by approximation scheme:
Lva (Cat(0)), Cat(O)) Vict(o) ar (w, Cf), Cat( Cat())).
(5)
where w and a indicate the current weights of the student and the learning rate of categor- ical distribution, respectively. In particular, we employ w with a single-step adapting (i.e., a Ltraii (w, (0)) to appropriate w² (Cat(O)) for avoiding the inner optimization in Eq. This appropriation scheme has been widely used meta-learning (Finn al., 2017) and neural architecture search (Liu et al., 2019a).
However, in the case of CR, there a consistency gap between the categorical distribution optimization and evaluation in terms of the teacher's logits. For categorical distribution op- timization, Fr,. (z; Cat(f)) is used to compute the difference between the student's logits as Evex LalTT, .... (o:; Cat(O)), fs(x;; w)). For categorical distribution evaluation, however, only the logits of the sampled teacher fiet... (x:) is used to obtain the difference to the student's 's logits as Exex falSTeTi,, (x), fs( w)).
4

---

Under review as a conference paper at ICLR 2023
To alleviate the consistency gap, propose SSWO whose objective function be written ((w; 0)
Σ (t) (s);
where i plays also role like label smoothing (Szegedy et al., 2016) which aims to reduce the confidence coefficient of the sampled teacher and avoid over fitting (Müller et al., 2019) of the cate- gorical distribution. Moreover, the smaller the sampling weight, the more reduction the confidence coefficient of the sampled teacher. Subsequently, the sampled single-weight can be optimized by
V (d Leal (w" (0), = V Ca-Cas feat ww (w, 0),0)
(7)
In practice, the proposed SSWO achieves better performance than CR, as shown in Section 4.2.
2.4.2 TEACHER SELECTION
After phase-1 optimization completion, ineffective teachers are separated from the teacher team according to the current categorical distribution Cat(o()), where the smaller the weight, the more ineffective the teacher. For avoiding categorical distribution optimizing from scratch, we present teacher selection strategy which assigns the weights of m ineffective teachers to n m effective teachers, to deliver the categorical distribution Cat( for phase- optimization by
mask(() smallest(Cat((0)),,. m))
(8) where p (1 in this paper) denotes the exponent value in the formulation, is small value (le-12 in this paper) to avoid division by zero, musmallest (Cat((())) m) obtains indexes of ineffective teachers according to Cat(o()), and mask(.) generates mask where the values of m ineffective and n - m effective teachers are set to and 1, respectively.
Cat(O(
m))
EXPERIMENTS AND RESULTS
3.1 DATASETS AND SETTINGS
Datasets. We evaluate the proposed AutoSKDBERT on GLUE benchmark (Wang et al., 2019), including MRPC (Dolan & Brockett, 2005), RTE (Bentivogli al., 2009), CoLAA (Warstadt al., 2019), SST-2 (Socher et al., 2013), QQP (Chen et al., 2018), QNLI (Rajpurkar et al., 2016) and MNLI (Williams et al., 2017). Moreover, STS-B (Cer al., 2017) not selected.
Settings. We employ the development set of GLUE benchmark dubbed as GLUE-dev, for cate- gorical distribution evaluation of AutoSKDBERT. We employ teacher team which consists of 14 BERT-style teachers, to distill a 6-layer BERT-style student dubbed AutoSKDBERT. The architec- ture information of the student and the teachers can be found in Appendíx C.I. On the one hand, we employ weak Toi to Toe (refer to Table 12) to verify a guess that the diversities of those weak teachers contribute to improve the distillation performance clusion that the extreme strong teacher (i.e., Ti3 and T14) can not always contribute to improving the distillation performance (see Appendix B.4 and F in the revised manuscript), we employ strong T13 and Ti4 verify the effectiveness of the proposed distillation paradigm for capacity gap alleviation. We give general way to design the teacher team and determine the value of ra Appendix A.
not. On the other hand, under a con-
3.2 TWO-PHASE OPTIMIZATION
We employ identical experimental settings for student distillation and categorical distribution opti- mization in both phase-l and phase-2 optimization. The original training set is split fifty-fifty into two subsets, i.e., training subset for student distillation (see Section 2.3) and validation subset for categorical distribution optimization (see Section 2.4).
3.2.1 STUDENT DISTILLATION
We choose Adam with weight decay of lc-4 the optimizer for student distillation. For various downstream tasks, we employ different batch size, learning rate and epoch number as shown in Table 2. Other hyree-parameters can be found in Appendix C.2.
5

---

Under review as a conference paper at ICLR 2023
Table 2: The sypper-arameters. for student distillation.
Flyper-parameter
MRPC
RTE
CoLA
SST-2
QQP
QNLI
MNLI
Batch Size
32
16
64
32
32
32
Learning Rate
le-5
le-5
le-5
le-5
3e-5
2c-5
2c-5
Epoch Number
50
50
2
10
3.2.2 CATEGORICAL DISTRIBUTION OPTIMIZATION
For categorical distribution optimization, we employ other Adam with weight decay of 1e-3 the optimizer. There are two important hyper-parameters: 1) the number of the ineffective teacher and 2) learning rate for categorical distribution optimization. Similarly, for different downstream tasks, the above two parameters are various as shown in Table 9. Other hyper-pane parameareeterss are identical to student distillation. The impact of each hyper-parameter is discussed in Appendix E.
Table 3: The hyper- parameters for categorical distribution optimization.
Hyper-parameter
MRPC
RTE
CoLA
SST-2
QQP
QNLI
MNLI
Ineffective Teacher Number
9
10
Learning Rate
9c-4
le-3
7c-4
9c-4
le-3
6e-4
4e-4
3.3 CATEGORICAL DISTRIBUTION EVALUATION
3.3.1
IMPLEMENTATION DETAILS
AutoSKDBERT delivers 25 categorical distribution candidates in phase-2 optimization, and trains the student with 25 candidates from scratch to choose the optimal categorical distribution. In addi- tion epoch number, other hyper-parameters (c.g., batch size, learning rate, etc.) are identical to student distillation on various downstream tasks as shown in Table 2. The epoch number is set to 15 on MRPC, RTE, CoLA tasks, and on SST-2, QQP, QNLI and MNLI tasks.
3.3.2 LEARNED CATEGORICAL DISTRIBUTION
We show the categorical distributions learned by AutoSKDBERT on GLUE benchmark in Figure 2. Each teacher model shows various importances on different downstream tasks. 1) The strongest teacher T14 plays domi- nant role on CoLA, SST-2 and QNLI tasks. 2) Low-capacity teachers, e.g..
0.00
UNLI OQP SSF-2 CoLA RITE MRPC
FLIV
0.0716
0.0
0.0
0.0
6.0
0.0
0.179
0.2002
0.0
0215002217
02127
0.0
0.0
0.0
0.0
9.0
0.0
6,995
01437
0L0099
DU7M5
0.117
0.1297
0.09
0UST6
3.307
To2 to Tos, can also provide useful knowledge for student distillation MRPC, CoLA and QNLI tasks. 3) The capacity of the effective teacher
0,0
0.0
6.0
0A
0.243
00
A12213
0.0
112214
0.6
0.0
0.0
6.0
11132
1:977
not always larger than the dis-
0.382 6,888
0.654
00436
card teachers RTE, SST-2 and
QQP tasks. Moreover, the search and evaluation costs with respect each downstream task are shown in Ap- pendix F.
0.134
Toi To2
Toa
Toa
Tos
Tos
To7
Tos
Tos
T10
Tu
T12
Tι3
T1a
Figure 2: Categorical distributions Iearned GLUE benchmark.
3.4 RESULTS AND ANALYSIS ON GLUE BENCHMARK
Table 4 summarizes the performance of AutoKKDBERT and the comparative approaches GLUE- dev. The proposed AutoSKDBERT achieves sfute-o-thhe-eee art performance on four out of seven tasks. AutoSKDBERT contributes to achieving better performance those tasks with small data size, e.g., MRPC and RTE. On MRPC, AutoSKDBERT achieves 93.2 Fl score and 90.7 accuracy score which 0.6 and 1.2 point higher than previous state-of-the-art MoEBERT (Zuo et al., 2022), respectively.

---

Under review as a conference paper at ICLR 2023
Table 4: Results of AutoSKDBERT and other popular approaches GLUE-dev. All comparative ap- proaches have identical architecture, i.e. 6-layer BERT-style language model with million parameters. and $ indicate that the results are cited from Xu al. (2020) and Zuo et al. (2022), respectively. means that the comparison between TinyBERT, & and AutoSKDBERT may not be fair since the former employs GloVe word embedding (Pennington al., 201 4) based data augmentation and transformer-lay layer distil- lation and embedding-layer distillation. indicates that the result is obtained by our settings with the distillation loss described in Wu et (2021), H. Moreover, the stronger teacher can not always contribute to improving the distillation performance of other approaches due to the capacity gap (Mirzadeh al., 2020) shown in Appendix G. Besides, show also the performances of multi-teacher AvgKD and TAKD with Tor to T14 in Appendix C.4.
experimental details can found Appendix
MRPC
RTE
CoLA
SST-2
QQP
QNLI
MNLI
Model
FI/acc
acc
Mce
acc
FI/acc
acc
m
Poor Man's BERTs (Sajjad al., 2020)
-/80.2
65.0
90.3
-/90.4
87.6
81.1
DistilBERT, (Sanh et al., 2019)
87.5/-
59.9
51.3
92.7
-888.5
89.2
82.2
LayerDrop (Fan et al., 2020)f
85.9/-
65.2
45.4
90.7
-/88.3
88.4
80.7
BERT-PKD (Sun et al., 2019)f
85.7/-
66.5
45.5
91.3
-/88.4
88.4
81.3
BERT-of-Thesee (Xu al., 2020)
89.0/-
68.2
51.1
91.5
-/89.6
89.5
82.3
MiniLMy1 (Wang et al., 2020)
88.4/-
71.5
49.2
-/91.0
91.0
84.0
MiniL.v22 (Wang ct al., 2020)
88.9/-
72.1
52.5
92.4
-/91.1
90.8
84,2
TinyBERTs (Jiao ct al., 2020)*
90.6/89.3
73.4
54.0
93.0
88.0/91.1
91.1
84.5
TinyBERT's (w/o aug) (Jiao al., 2020);
88.4/-
72.2
42.8
91.6
-/90.6
90.5
83.5
MT-BERT (Wu al., 2021)$
90.8/87.0
72.2
49.1
92.2
87.1/90.4
91.4
83.8
MoEBERT (Zuo et al., 2022)
92.6/89.5
74.0
55.4
93.0
88.4/91.4
84.5
AutoSKDBERT (Ours)
93.2/90.7
76.9
51.8
93.0
88.0/91
91.6
84.3
On the other hand, compared TinyBERT (Jiao al., 2020) and MOEBERT (Zuo et al., 2022) on RTE task, AutoSKDBERT achieves 3.5 and 2.9 point absolute improvement, respectively.
However, CoLA, TinyBERT and MoEBERT achieve 2.2 and 3.6 point absolute improvement compared to AutoSKDBERT, respectively. On the one hand, TinyBERT employs data augmenta- tion and transformer layer distillation to achieve high performance. On the other hand, MoEBERT employs 1) more complex student whose architecture is an ensemble of multiple experts, and 2) extra distillation procedure, i.e., transformer layer distillation, to achieve novel performance.
The proposed approach is general KD paradigm for BERT compression. Consequently, we imple- ment also extensive experiments to verify the effectiveness for image classification on CIFAR-100 (see Appendix B) and the orthogonality with other approaches (see Appendix D).
4 ABLATION STUDIES
4.1 TWO-PHASE OPTIMIZATION: PHASE- I VERSUS PHASE-2
In this section, AutoSKDBERT delivers also 25 categorical distribution candidates in phase-1 opti- mization. Subsequently, each categorical distribution candidate is trained from scratch using identi- cal settings described in Section 3.3, and the best-performing one on each task is shown in Table
Table 5: The performance of AutOSKDBERT with the best categorical distribution learned in phase-1 and phase-2 optimization on GLUE-dev.
Task
MRPC
RTE
CoLA
SST-2
QQP
QNLI
MNLI
Metries
Fl/aec
Mce
F1/ace
acc
m
Phase-1
92.9/90.2
73.7
49,2
92.9
87, 87.6/90.7
91.3
83.0
Phase-2
93.2/90.7
76.9
51.8
93.0
88.0/91.0
91.6
84.3
Gain
+0.3/+0.5
+1.4
+2.7
+0.1
+0.4/+0.3
+0.3
+1.3
Phase-2 optimization achieves better performance than phase-1 optimization, e.g., the absolute im- provement is more than 1.3 RTE. CoLA and MNLI, where those teachers weaker than the student are prone to providing useless knowledge even noise disturbance. However, low-capacity teachers contribute improving the performance of AutoSKDBERT MRPC, SST-2 and QNLI.

---

Under review as a conference paper at ICLR 2023
4.2 CATEGORICAL DISTRIBUTION UPDATE STRATEGY: CR-BASED VERSUS SSWO
ESWO-BASEL BASED
In Section 2.4.1, we propose SSWO which stochastically samples wingle-weight to optimize the categorical distribution, to alleviate the consistency gap between the categorical distribution opti- mization and evaluation of CR in terms of teachers' logits. For AuRRSKDBERT with CR. the used nyper-parameters of categorical distribution optimization and evaluation are identical to AutoSKD- BERT with SSWO, described in Section 3.2 and Section 3.3.
The proposed SSWO achieves better performance than CR on all tasks as shown in Table 6, Par- ticularly, the absolute improvement more than 1.6 point on RTE and CoLA tasks. Compared to Table 5, AutoSKDBERT with CR achieves higher performance than phase- AutoSKDBERT on six out of seven tasks. Consequently, useless teachers lead to more performance degradation than the consistency gap issue.
Table 6: The performance AutoSKDBERT with the hest categorical distribution learned by CR and SSWO for categorical distribution optimization on GLUE-dev.
Task
MRPC
RTE
CoLA
SST-2
QQP
ONLI
MNLI
Metries
Fl/acc
acc
Mec
acc
Fl/ace
acc
CR
92.9/90.2
74.7
50.2
92.7
87.9/91.0
91.5
83.9
SSWO
93.2/90.7
76.9
51.8
93.0
88.0/91.0
91.6
84.3
In addition to SSWO, heuristic optimization algorithms like evolutionary algorithm and reinforce- ment learning can also used to determine the categorical distribution. In this paper, we choose the most efficient one, i.e., gradient-based -based SSWO.
4.3 CATEGORICAL DISTRIBUTION GENERATION: RANDOM VERSUS LEARNING
We compare two groups of implementation of AutoSKDBERT with various algorithms for categor- ical distribution generation, i.e., random and learning. on GLUE-dev. For random algorithm, 200 categorical distributions are randomly generated for all teacher candidates. For learning algorithm, we employ different learning rates 3e-4 le-3 with an interval of le-4 for categorical distribu- tion optimization. Moreover, the ineffective teacher number is identical to Section 3.3.2 various tasks. Subsequently, each implementation delivers 25 categorical distributions in phase-2 optimiza- tion. Consequently, 200 categorical distributions are obtained. The comparison between random and leaning algorithms is shown in Figure
Nonher Distriurtion
Number of Distributions
Number of Distribrtons
Number of Distributios
40
80
40
40
20
75.0
Accuracy MRPC
Accuracy
Accuracy
Accuracy
Number of Distibustons
Distribarions
Number Distribrtins
150
150
100
75
100
Random
50
carning
50
50
25
$8.5
50.0
59.5
90.0
10.5
91.0
91.5
30
$4
Accuracy QQP
ONLI
Accuracy
MNLI
Figure 3: Comparison of AutOSKDBERT with random and learning algorithm for categorical distribution generation on GLUE-dev. Two types of algorithm are evaluated by 200 categorical distributions. MRPC and QQP tasks evaluated by the average of Fl score and accuracy score, CoLA task is evaluated by Matthews correlation coefficient, and other tasks are evaluated by accuracy score. Best viewed color.
8

---

Under review as a conference paper at ICLR 2023
The categorical distribution generation algorithm aims to achieve more high- hegh-performance Au- toSKDBERTs. As shown in Figure 3, the proposed learning algorithm contributes to obtaining better categorical distribution than those randomly generated each downstream task. Particularly, the learning algorithm plays a dominant role on MRPC, CoLA, QQP, QNLI and MNLI tasks. The best accuracy scores of random algorithm are 89.29 QQP and 83.37 MNLI, respectively. For the proposed learning algorithm, the worst accuracy scores are 89.11 on QQP and 83.44 on MNLI which rank the top 10 and the best in random algorithm, respectively.
RELATED WORK
5.1 PRE-TRAINED LANGUAGE MODEL
Based on the transformer-style architecture (Vaswani et al., 2017). BERT (Devlin et al., 2019) achieves state-of- the-hhhtart performance different natural language understanding benchmarks, e.g., GLUE (Wang et al., 2019), SQuAD (Rajpurkar et al., 2016; 2018). Subsequently, great number of variants of BERT proposed, e.g-. XLNet (Yang et al., 2019), ELECTRA (Clark et al., 2020) with new pre-training objectives, RoBERTa (Liu et al., 2019b), T5 (Raffel et al., 2020) with larger pre-training corpus, CORyBERT (Jiang et al., 2020) with various architectures and Synthesizer (Tay et al., 2020) with developed transformer-lit like block w.r.t. the dot-product self-attention mechanism. Besides, previous pre-trained language models often have several hundred million parameters (e.g. 335 million of BERT (Devlin et al., 2019), even 175 billion of GPT-3 (Brown et al., 2020)) which contribute to delivering amazing performance on downstream tasks while exponentially in- creasing the difficulty of deployment on resource-constraing device. ALBERT (Lan et al., 2020) adopts parameter sharing strategy to reduce the parameters, and achieves competitive performance.
5.2 KNOWLEDGE DISTILLATION FOR BERT-STYLE LANGUAGE MODEL. COMPRESSION
In order to obtain drvenll BERT-style language model, many KD-based compression ap- proaches have been proposed. DistilBERT (Sanh al., 2019) compresses a smaller, faster, cheaper and lighter 6-layer BERT-style language model via learning the soft target probabiliiis of the teacher in the pre-training stage. Sun al. (2019) propose patient knowledge distillation which transfers knowledge from the last every layers, to compress BERT-style language model in the fine-tuning phase. In MobileBERT (Sun et al., 2020), an inverted- -ootteeebbbt BERT-style language model is pre- trained transfer knowledge
lask-agnostic MobileBERT in layer-to-layer way. The student in MiniLM (Wang et al., 2020) imitates not only the attention distribution of the teacher, but also the deep self-attention knowledge which reflects the difference between values. In both the pre-training and the fine-tuning phases, TinyBERT (Jiao et al., 2020) learns various knowledge from hidden layer, final layer, embedding and self-attention to achieve high performance. Moreover, Glo Ve word embedding (Pennington et al., 2014) based data augmentation technique is employed to further im- prove the performance of Tiny MT-BERT (Wu et al., 2021) employs multiple teachers to achieve better performance than single-teacher KD based approaches on several downstream tasks.
6 CONCLUSION
This work proposes AutoSKDBERT, which is a new paradigm of knowledge distillation for BERT model compression. teacher is stochastically sampled from a predefined multi-level teacher team in each step to distill the student following a categorical distribution. We observe that the categorical distribution plays an important role for obtaining perforrrfoee AutoSKDBERT. Consequently, we propose two-phase optimization framework to learn the best categorical distribution via SSWO. The first phase distinguishes effective teachers from ineffective teachers. In the second phase, the effective teachers are further optimized. Moreover, before phase-2 optimization beginning, the in- effective teachers are discarded and their weights are assigned to the effective teachers via teacher selection strategy. Extensive experiments GLUE benchmark show that the proposed AutoSKD. BERT achieves state-of-the- performance compared to popular compression approaches on sev- eral downstream tasks.

---

Under review as a conference paper at ICLR 2023
REFERENCES
Sungsoo Ahn, Shell Xu Hu, Andreas Damianou, Neil D Lawrence, and Zhenwen Dai. Variational information distillation for knowledge transfer. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR, pp. 9163-9171, 2019.
Luisa Bentivogli, Peter Clark, Ido Dagan, and Danilo Giampiceolo. The fifth pascal recognizing textual entailment challenge. In TAC, 2009.
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. In Advances in Neural Information Processing Systems (NeurlPS), volume 33, pp. 1877-1901, 2020.
Daniel Cer, Mona Diab, Eneko Agirre, Inigo Lopez-Gazpio, and Lucia Specia. Semeval-2017 task I: Semantic textual munillartyy-ultilingee and cross-lingual focused evaluation. In Proceedings of the 11th Inter- national Workshop on Semantic Evaluation, 2017.
Zihan Chen, Hongbo Zhang, Xiaoji Zhang, and Leqi Zhao. Quora question pairs. University of Waterloo, pp. 1-7, 2018.
Kevin Clark, Minh-Than Thang Luong, Quoe V Le, and Christopher D Manning. ELECTRA: Pre-training text encoders as discriminators rather than generators. In 8th International Conference on Learn ing Representations, ICLR, 2020.
Benoît Colson, Patrice Marcotte, and Gilles Savard. An overview of bilevel optimization. Annals of operations research, 153()))23--256, 2007.
Jacob Devlin, Ming-Wer Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACL-IILT, volume 1, pp. 4171-4186, 2019.
Bill Dolan and Chris Brockett. Automatically constructing corpus of sentential paraphrases. In Third International Workshop on Paraphrasing, IWP, 2005.
Yihe Dong, Jean- BanntBapistte Cordonnier, and Andreas Loukas. Attention is not all you need: Pure attention loses rank doubly exponentially with depth. In International Conference on Machine Learning, pp. 2793-2803. PMLR, 2021.
Angela Fan, Edouard Grave, and Armand Joulin. Reducing transformer depth on demand with structured dropout. In 8th International Conference on Learning Representations, ICLR, 2020.
Chelsea Finn, Pieter Abbeel, and Sergey Levine. Model-agnostic meta-learning for fast adaptation of deep networks. International Conference on Machine Learning (ICML), pp. 1126-1135. PMLR, 2017.
Jiahui Gao, Hang Xu, Han Shi, Xiaozhe Ren, LH Philip, Xiaodan Liang, Xin Jiang, and Zhenguo Li. AutoBERT-zero: Evolving bert backbone from scratch. Proceedings of the AAAI Conference on Artificial Intelligence, volume 36, pp. 10663-10671, 2022.
Mitchell A. Gordon, Kevin Duh, and Nicholas Andrews. Compressing BERT:
studying the effects of weight pruning on transfer learning. In Proceedings of the 5th Workshop on Representation Learning for NLP, pp. 143-155, Association for Computational Linguistics, 2020.
Fuming Guo, Sijia Liu, Finlay S Mungall, Xue Lin, and Yanzhi Wang. Reweighted proximal pruning for large-scale language representation. arXiv preprint arXiv:1909, 12486, 2019.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In Proceedings of the IEEE conference on Computer Vision and Pattern Recognition, CVPR, pp. 770-778, 2016.
Byeongho Heo, Minsik Lee, Sangdoo Yun, and Jin Young Choi. Knowledge transfer via distillation of activation boundaries formed by hidden neurons. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 33, pp. 3779-3787, 2019.
10

---

Under review as a conference paper at ICLR 2023
Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. Distilling the knowledge in neural network. arXiv preprint arXiv:1503.0253) 2015.
Zehao Huang and Naiyan Wang. Like what you like: Knowledge distill via neuron selectivity transfer. arXiv preprint 1707:1707.00211 2017.
Forrest N landola, Albert E Shaw, Ravi Krishna, and Kurt W Keutzer. Squeezebert: What can computer vision teach nip about efficient neural networks? arXiv preprint arXiv:2006. 1316, 2020.
Zi- Zi-Hang Jiang, Weihao Yu, Daquan Zhou, Yumpeng Chen, Jiashi Feng, and Shuicheng Yan. Con- vbert: Improving bert with span based dynamic convolution. In Advances in Neural Information Processing Systems (NeurIPS), volume 33, pp. 12837-12848, 2020.
Xiaoqi Jiao, Yichun Yin, Lifeng Shang. Xin Jiang, Xiao Chen, Linlin Li, Fang Wang, and Qun Liu. Tinybert: Distilling BERT for natural language understanding. In Proceedings of the 2020 Conference on Empirical Methods Natural Language Processing, EMNLP, pp. 4163-4174, 2020.
Jangho Kim, SkongUk Park, and Nojun Kwak. Paraphrasing complex network: Network com- pression via factor transfer. In Advances in Neural Information Processing Systems, NeurIPS, volume 31, 2018.
Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. 2009.
Zhenzhong Lan, Mingda Chen, Sebastian Goodman, Kevin Gimpel, Piyush Sharma, and Radu Sori- cut. ALBERT: A lite BERT for self-supervised learning of language representations. In 8th International Conference on Learning Representations, ICLR, 2020.
Hector Levesque, Ernest Davis, and Leora Morgenstern. The winograd schema challenge. In Thir- teenth international conference on the principles of knowledge representation and reasoning. 2012.
Hanxiao Liu, Karen Simonyan, Yiming Yang, et al. DARTS: Differentiable architecture search. In International Conference on Learning Representations (ICLR), 2019a.
Yinhan Liu, Myle Out, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. RoBERTa: A robustly optimized BERT pre- training approach. arXiv preprint arXiv:1907.11662, 2019b.
Paul Michel, Omer Levy, and Graham Neubig. Are sixteen heads really better than one? In Advances in Neural Information Processing Systems, NeurIPS, volume 32, 2019.
Seyed Iman Mirzadeh, Mehrdad Farajtabar, Ang Li, Nir Levine, Akihiro Matsukawa, and Hassan Ghasemzadeh. Improved knowledge distillation via teacher assistant. In Proceedings the the AAAI Conference on Artificial Intelligence, AAAI, volume 34, pp. 5191-5198. 2020.
Rafacl Müller, Simon Kornblith, and Geoffrey E Hinton. When does label smoothing help? In Advances in Neural Information Processing Systems (NeurIPS), volume 32, 2019.
Haojie Pan, Chengyu Wang, Minghui Qiu, Yichang Zhang, Yaliang Li, and Jun Huang. Meta-KD: A meta knowledge distillation framework for language model compression across domains. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the IIth International Joint Conference on Natural Language Processing, ACI/IJCNLP, pp. 3026- 3036, 2021.
Wonpyo Park, Dongju Kim, Yan Lu, and Minsu Cho. Relational knowledge distillation. In Pro- ceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR, pp. 3967-3976, 2019.
Nikolaos Passalis and Anastasios Tefas. Learning deep representations with probabilistic knowledge transfer. In Proceedings of the European Conference on Computer Vision, ECCV, pp. 268-284, 2018.
11

---

Under review as a conference paper at ICLR 2023
Baoyun Peng, Xiao Jin, Jiaheng Liu, Dongsheng Li, Yichao Wu, Yu Liu, Shunfeng Zhou, and Zhaoning Zhang. Correlation congruence for knowledge distillation. In Proceedings of the IEEE/CVF International Conference Computer Viston ICCV, pp. 5007-5016,, 2019.
Jeffrey Pennington, Richard Socher, and Christopher D Manning. GloVe: Global vectors for word representation. In Proceedings of the 2014 Conference on Empirical Methods in Natural Lan- guage Processing, EMNLP, pp. 1532-1543, 2014.
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. Exploring the limits of transfer learning with a unified text-to-text transformer. J. Mach. Learn. Res., 21:1 40:1-440:67, 2020.
Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Perey Liang. Squad: 100, 000+ questions for machine comprehension of text. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing. EMNLP, pp. 2383 2392, 2016.
Pranav Rajpurkar, Robin Jía, and Percy Liang. Know what you don't know: Unanswerable questions for squad. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics, ACL, pp. 784-789, 2018.
Adriana Romero, Nicolas Ballas, Samira Ebrahimi Kahou, Antoine Chassang, Carlo Gatta, and Yoshua Bengio. Fitnets: Hints for thin deep nets. In 3th International Conference on Learning Representations, ICLR, 2015.
Hassan Sajjad, Fahim Dalvi, Nadir Durrani, and Preslav Nakov. Poor man's bert: Smaller and faster transformer models. arXiv preprint arXiv:2004.03844 2020.
Victor Sanh, Lysandre Debut, Julien Chaumond, and Thomas Wolf. DistilBERT, a distilled version of BERT: Smaller, faster, cheaper and lighter. arXiv preprint 100001191, 2019.
Sheng Shen, Zhen Dong, Jiayu Ye, Linjian Ma, Zhewei Yao, Amir Gholami, Michael W Mahoney, and Kurt Keutzer. Q-BERT: Hessian based ultra low precision quantization of BERT. In Pro- ceedings of the AAAI Conference Artificial Intelligence, volume 34, pp. 8815-8821, 2020.
Han Shi, Jiahui Gao, Xiaozhe Ren, Hang Xu, Xiaodan Liang, Zhenguo Li, and James Tin-Yat Kwok. Sparsebert: Rethinking the importance analysis in self-attention. In International Confer- ence on Machine Learning, ICML, pp. 9547-9557. PMLR, 2021.
Richard Socher, Alex Perelygin, Jean Wu, Jason Chuang, Christopher Manning, Andrew Ng, and Christopher Potts. Recursive deep models for semantic compositionality over a sentiment treebank. In Pruceedings of the 2013 Conference on Empirical Methods in Natural Language Processing, EMNLP, pp. 1631-1642, 2013.
Siqi Sun, Yu Cheng, Zhe Gan, and Jingjing Liu. Patient Knowledge Distillation for BERT Model Compression. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing, EMNLP- IJCNLP, pp. 4322-4331, 2019.
Zhiqing Sun, Hongkun Yu, Xiaodan Song, Renjie Liu, Yiming Yang, and Denny Zhou. Mobilebert: A compact task-agnostic BERT for resource-limited devices. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, ACL, pp. 2158-2170 2020.
Christian Szegedy, Vincent Vanhoucke, Sergey loffe, Jon Shlens, and Zbigniew Wojna. Rethink- ing the inception architecture for computer vision. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 2818-2826, 2016.
Tay, D Bahri, Metzler, Juan, Zhao, and C Zheng. Synthesizer: Rethinking self-attention in transformer models. arxiv 2020. arXiv preprint arXiv:2005.00743, 2, 2020.
Yonglong Tian, Dilip Krishnan, and Phillip Isola. Contrastive representation distillation. In 8th International Conference Learning Representations, ICIRR, 2020.
12

---

Under review as a conference paper at ICLR 2023
Linh Tran, Bastiaan S Veeling, Kevin Roth, Jakub Swiatkowski, Joshua V Dillon, Jasper Snoek, Stephan Mandt, Tim Salimans, Sebastian Nowozin, and Rodolphe Jenatton. Hydra: Preserv- ing ensemble diversity for model distillation. In International Conference on Machine Learning Workshop on Uncertainty and Robustness in Deep Learning, 2020.
Frederick Tung and Greg Mori. Similarity IEEE/CVF International Conference on Computer Vision, ICCV, pp. 1365-1374, 2019.
knowledge distillation. In Proceedings of the
Iulia Ture, Ming-We Wei Chang, Kenton Lee, and Kristina Toutanova. Well-read students learn better: The impact of student initialization on knowledge distillation. arXiv preprint wrXiv:190..060 2019.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. volume 30, 2017.
Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R Bowman. GLUE: A multi-task benchmark and analysis platform for natural language understanding. In 7th International Conference on Learning Representations, ICLR, 2019.
Wenhui Wang, Furu Wei, Li Dong, Hangho Bao, Nan Yang, and Ming Zhou. MiniL Deep self- attention distillation for task-agnostic compression of pre-trained transformers. In Advances in Neural Information Processing Systems, NeurlPS, volume 33, pp. 5788, 2020.
Alex Warstadt, Amanpreet Singh, and Samuel R Bowman. Neural network acceptability judgments. Transactions of the Association for Computational Linguistics, 7:625-641, 2019.
Adina Williams, Nikita Nangia, and Samuel R Bowman. broad-coverage challenge corpus for sentence understanding through inference. In Proceedings of the 2017 Conference of the North American Chapter of the Association for Computational Linguistics, NAACI, 2017.
Chuhan Wu, Fangzhao Wu, and Yongfeng Huang. One teacher enough? pre-trained language model distillation from multiple teachers. In Findings of the Association for Computational Lin- guistics: ACL/JCNLP,, pp. 4408 4413, 2021.
Canwen Xu, Wangchunshu Zhou, Tao Ge, Furu Wei, and Ming Zhou. BERT-of-Theseus: Compress- ing BERT by progressive module replacing. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing, EMNLP, pp. 7859-7869, 2020.
Zhilin Yang, Zihang Dai, Yiming Yang. Jaime Carbonell, Russ R Salakhutdinov, and Quoe V Le. XLNet: Generalized autoregressive pretraining for language understanding. Advances in Neu- ral Information Processing Systems, NeurlPS, volume 32, 2019.
Junho Yim, Donggyu Joo, Jihoon Bae, and Junmo Kim. A gift from knowledge distillation: Fast optimization, network minimization and transfer learning. In Proceedings of the IEFE Conference on Computer Vision and Pattern Recognition, CVPR, pp. 4133-4141, 2017.
Sergey Zagoruyko and Nikos Komodakis. Wide residual networks. In Proceedings of the British Machine Vision Conference 2016, BMVC, 2016.
Sergey Zagoruyko and Nikos Komodakis. Paying more attention to attention: Improving the perfor- mance of convolutional neural networks via attention transfer. 2017.
Simiao Zuo, Qingru Zhang, Chen Liang, Pengeheng He. Tuo Zhao, and Weizhu Chen. MoEBERT:
from BERT to of via importance-guiide adaptation. In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACL, pp. 1610-1623, 2022.
13

---

Under review as a conference paper at ICLR 2023
A INEFFECTIVE TEACHER NUMBER DETERMINATION AND TEACHER TEAM DESIGN
A.1 INEFFECTIVE TEACHER NUMBER DETERMINATION
A simple way to choose the ineffective teacher number is elaborately designing the teacher team and setting m to the number of weak teachers whose capacities are weaker than student. Moreover, the student itself can be treated the above weak teacher.
A.2 TEACHER TEAM DESIGN
First, we should determine the strongest teacher and student. Next, select several teacher assis- tants whose capacities are stronger than student but weaker than the strongest teacher. Finally, we choose also several weak teachers whose capacities are weaker than student. Above all, the pre- defined teacher team consists of several weak teachers, several teacher assistants and the strongest teacher.
B AUTOSKD FOR IMAGE CLASSIFICATION
To verify the effectiveness of the proposed distillation paradigm computer vision, conduct three groups of experiment on CIFAR-100 (Krizhevsky et al., 2009) image classification dataset. Following CRD (Tian et al., 2020), choose three student models: 1)WRN-16-2, WRN-40-1 and 3) ResNet-8 x4. WRN-d-w represents Wide ResNet with depth d and width factor w. ResNet- 4 indicates 4 times wider network (namely, with 64, 128, and 256 channels for each block) with depth d. Moreover, in CRD, the above student models are distilled by WRN-40-2, WRN-40-2 and ResNet-32 44, respectively.
B.1 DATASET
As a popular dataset for image classification, CIFAR-100 consists of 60000 images (50000 for training and 10000 for test) with 32x32 pixels. Similar the experiment for BERT compression, the original training set is split fifty-fifty into two subsets, i.c., training subset for student distillation and validation subset for categorical distribution optimization.
B.2 DETAILS OF TEACHER TEAM
For various student models, we select different teacher teams according to Appendix A.2 as shown in Table 7.
Table 7: Details teacher team each student model.
Student
Teacher Team
WRN-16-2
WRN-16-2, WRN-22-2, WRN-28-2, WRN-34-2, WRN-40-2
WRN-40-1
WRN-40-1, WRN-16-2, WRN-22-2, WRN-28-2, WRN-34-2, WRN-40-2 ResNet-8xx, ResNet- 14 44, ResNet-20x4, ResNet-26x4, ResNet-32x4
R4
Moreover, the performance of each teacher model on CIFAR-100 is shown in Table 8.
Table 8: Performance of each teacher model on CIFAR-100.
Teacher
WRN-40-1
WRN-16-2
WRN-22-2
WRN-28-2
WRN-34-2
WRN-40-2
Accuracy
71.19
73.18
76.19
76,21
76.14
75.61
Teacher
ResNet-8x4
1444
ResNet-20x4
ResNet-26x4
ResNet-32x4
Accuracy
72.85
76.90
77.96
78.75
79.42
14

---

Under review as a conference paper at ICLR 2023
B.3 CATEGORICAL DISTRIBUTION OPTIMIZATION
Similar to the experiment for BERT compression, there are two hyper-parameters, i.e., ineffective teacher number and learning rate for categorical distribution optimization. According Appendix I. we fix ineffective teacher number to I, and choose categorical distribution learning rate from 3e- 4 to 1e-3 with an interval of le-4 for three groups of experiment. Different from BERT compression, we choose SGD as the optimizer with CosineAmeallng learning rate scheduler, initial learning rate of 0.05 and batch size 64 for student model training. Moreover, the number of epochs is set 50, and later 25 epochs deliver 25 categorical distribution candidates.
Table 9: The hyper- parameters for categorical distribution optimization.
Hyper-parameter
WRN-16-2
WRN-40-1
ResNet-8xx
Ineffective Teacher Number
Learning Rate
le-3
lc-3
4e-4
B.4 CATEGORICAL DISTRIBUTION EVALUATION
Following the experimental settings in CRD (Tian al., 2020), we train the student model with 25 categorical distribution candidates for 240 epochs, and employ SGD as the optimizer with batch size of 64, learning rate of 0.05 which decayed by factor 0. when arriving 1500th, 180-th, 210-th epoch and weight decay of 5e-4.
B.5 LEARNED CATEGORICAL DISTRIBUTION
For various student models, Table 10 shows the learned categorical distributions. Similarly, for each student model, the weakest teacher model, i.e., student itself, is considered as the ineffective teacher model when = 1.
Table 10: Learned categorical distributions for various student models CIFAR-100.
Student
WRN-16-
Teacher WRN-16-2
WRN-22-2
WRN-28-2 28-2
WRN-34-2
WRN-40-2
Weight
0.2518
0.2513
0.2458
0.2511
Student
WRN-40-1
Teacher
WRN-40-1
WRN-16-2
WRN-22-2
WRN-28-2
WRN-34-2 0.2034
WRN-40-2
Weight
0.1864
0.2068
0.2011
0.2024
Student
Res
Teacher
ResNet-8x4
ResNet- 14x4
ResNet-20x4
Res
ResNet-32x4
Weight
0
0.2299
0.2473
0.2565
0.2663
B.6 RESULTS AND ANALYSIS
Following Tian et al. (2020), we show the test accuracy of the last epoch in Table 11 for a fair comparison. The proposed distillation paradigm achieves the best performance for 2 out of student models. Particularly, compared to previous state-of-the-an CRD, the improvements are 0.56% and 0.42% for WRN-16-2 and WRN-40-1 distillation, respectively.
C DETAILS OF STUDENT AND TEACHER TEAM FOR AUTOSKDBERT
C.I ARCHITECTURE INFORMATION
The architecture information of student and teachers shown in Table 12.
C.2 HYPER-PARE PARAMETERS FOR FINE-TUNING AND DISTILLATION
We utilize the hyper-parameters shown in Table 13 for fine-tuning and distillation.
15

---

Under review as a conference paper at ICLR 2023
Table 11: Test accuracy (%) of the proposed AutoSKD and other popular distilla- tion approaches CIFAR- 100. experimental results are cited from Tian et al. (2020). Average of the last epoch over runs.
Student
WRN-16-2
WRN-40-1
ResNet-8xx4
Teacher
WRN-40-2
WRN-40-2
ResNet-32x0
Student Accuracy
73.26
71.98
72.50
Teacher Accuracy
75.61
75.61
79.42
(Hinton et al., 2015)
74.92
73.54
73.33
FitNet (Romero al., 2015)
73.58
72.24
73.50
AT (Zagoruyko& & Komodakis, 2017)
74.08
72.77
73.44
SP (Tung & Mori, 2019)
73.83
72.43
72.94
CC (Peng al., 2019)
73.56
72.21
72.97
VID (Ahn et al., 2019)
74.11
73.30
73.09
RKD (Park et al., 2019)
73.35
72.22
71.90
PKT (Passalis & Tefas, 2018)
74.54
73.45
73.64
AB (Heo al., 2019)
72.50
72.38
73.17
FT (Kim al., 2018)
73.25
71.59
72.86
FSP (Yim et al., 2017)
72.91
72.62
NST (Huang & Wang, 2017)
73.68
72.24
73.30
CRD (Tian et al., 2020)
75.48
74.14
75,51
AutoSKD (Ours)
76.04
74.72
75.39
Table 12: The architecture of each student and teacher.
Model
Name
Layer
Hidden Size
Head
#Params (M)
Student
AutoSKDBERT
768
12
66.0
Toi
128
2
5.6
Toz
128
2
Tos
12
128
2
6.4
To4
256
4
14.3
Tos
10
256
4
15.9
To6
12
256
4
17.5
To7
41.4
Teacher
Tos
10
512
8
47.7
Too
12
512
8
54.0
T10
768
12
81.1
Tı1
10
768
12
95.3
T12
12
T13
24
1024
16
335
T14
24
1024
16
335
Pre-traiming with whole werd masking.
Table 13: Hlpper-paaameters for fine-tuning of student and teacher team.
Hyper-parameter
Value
Adam
Ic-6
Adam 31
0.9
Adam 3,
0.999
Learning rate decay
linear
Warmup fraction
0.1
Attention dropout
0.1
Dropout
Weight decay
le-4
Batch size
32 for fine-tuning, {16, 32) for distillation
For Tıa and T14. (6e-6, 7e-6, 8e-6, 9e-6) MRPC and RTE tasks, (2e-5, 3e-5, 4e-5, 5c-5) other tasks. For student and other teachers, {2c-5, 3c-5, 4e-5, 5c:5) and 1c--5 2c-5, 3c-5} on all tasks, respectively.
Learning
Fine-tuning epochs
15 on MRPC, RTE and CoLA tasks, on other tasks
16

---

Under review as a conference paper at ICLR 2023
C.3 FINE-TUNING PERFORMANCE
On the one hand, we directly treat the pre-trained model of TinyBERT as the student of AutoSKD. BERT. On the other hand, we choose 14 BERT-style language models with various capabilities as the candidates for teacher team. Moreover, each pre-trained teacher can be downloaded from official implementation of BERT Furthermore, the results of the student and the teacher on GLUUE-ev are shown in Table 14.
Table 14: The fine-tuning performances of student and teachers on GLUF-devv.
MRPC FIFsce
RTE
CoLA
SST-2
QQP Pieaac
QNLI
MNLI
Model
Avg
acc
Mcc
acc
acc
m
Student
89.44
71.84
45.74
91.63
86.44
90.70
82.55
85.87
Toi
81.83
66.06
25.92
86.24
83.95
83.80
72.95
80.00
To2
84.75
66.06
25.57
85.67
84.18
84.00
73.75
80.49
Toa
84.59
65.70
27.87
86.47
85.02
84.40
75.16
81.01
To4
85.18
64.62
40.35
89.33
86.36
86.80
78.16
82.46
Tos
87.84
66.06
38.76
89.33
87.25
87.26
78.75
83.42
Tos
85.96
66.06
41.36
89.68
87.21
87.42
79.54
83.27
Tor
87.91
70.04
48.14
91.28
88.69
89.27
80.84
85.21
Tos
88.17
65.70
50.98
91.28
88.62
89.25
81.41
84.74
Toa
88.85
66.43
$3.58
92.09
89.01
90.33
81.90
85.34
Tio
89.36
68.95
56.30
93.00
89.27
90.79
83.05
86.21
Tn1
90.10
71.12
60.32
92.78
89.71
91.20
84.00
86.93
T12
89.98
68.59
60.26
92.66
89.66
91.85
84.40
86.76
T13 90.60
62.74
62.74
94.50
90,26
92.70
86.88
86.83
T14
90.15
79.06
65.88
94.72
90.40
93.89
87.06
89,50
C.4 PERFORMANCE OF STUDENT WITH VARIOUS DISTILLATION PARADIGMS
Table 15 summarizes the performance of student using different distillation paradigms with the teacher models described in Appendix C.I. Moreover, experimental settings can be found in Ta- ble 13. On the one hand, the student performance using single-teacher distillation with respect to each teacher model is given. On the other hand, two popular multi-teache KD paradigms, i.e., AvgKI) (Hinton et al., 2015) and TAKD (Mirzadeh al., 2020), are employed to distill the student with two groups of teacher team, i.e., Toi to T14 and T10 to T14-
According to Table 15, we can draw several conclusions:
1. For single-teacher KD paradigm, the strongest teacher may not be the best teacher for stu- dent distillation. Capacity gap (Mirzadeh et al., 2020) between the strong-capacins teacher and weak-capacity student plays an important role for this phenomenon.
2. For multi-teacher AvgKD, increasing the number of teachers to improving the distillation performance. In AvgKD, the diversity losing issue leads to unsatisfactory performance due to using the ensemble of teacher outputs.
not always contribute
3. For multi-teacher TAKD, weak-capacity teachers dramatically reduce the distillation per- formance of student. In TAKD, the weakest teacher assistant (e.g., Toi for the teacher team To1-The Tı0 for the teacher team T14-T14) transfers mixture of knowledge which learned from previous stronger teacher assistants (e.g.. Toz to T14 for the teacher team Tor Toi-T14- Tn to T14 for the teacher team TT,4 into the student. As result, the performance of TAKD is very sensitive to the capacity of the weakest teacher assistant.
In order verify the effectiveness of weak-capacity teacher for performance improvement, choose several weak-capacity BERT-style models as teachers, e.g., Toi to Teg. Besides, we choose also two atron-ccapacty teachers. i.e., Tıs and Tıa in Table 12, to verify the effectiveness of the proposed distillation paradigm for capacity gap alleviation.
TinyBERT General. 61. 7680
'https:// thub. www. com/googleeee reseerachheeeeceeeeeeeeae
17

---

Under review as a conference paper at ICLR 2023
Table 15: Distillation performance of student with various distillation paradigms GLUB-dev.
MRPC
RTE
CoLA
SST-2
QQP
QNLI
MNLI
KD Paradigm
Teacher
Mcc
Toi
84.6
67.9
32.5
88.8
84.6
86.3
74.9
To2
87.9
67.2
35.3
89.7
85.1
86.2
75.3
Ta3
87.1
70.8
36.7
90.5
86.1
86.5
76.2
Tos
89.7
69.0
40.2
92.4
87.1
89.6
79.7
Tos
89.7
91.4
87.8
Toe
88.4
69.3
45.8
92.4
87.6
90.2
80.4
Tu7
89.5
73.7
48.3
92.8
88.6
91.0
81.9
Single-teacher
Tus
89.6
71.5
46.7
92.3
88.8
90.9
82.3
Tos
89.9
72.2
46.2
92.2
88.9
91.5
82.7
Tio
89.6
73.3
49.3
92.0
88.9
91.1
82.9
Tı1
89.7
71.8
48.5
92.3
89.0
91.3
83.2
T12
89.1
71.5
46.9
93.1
88.9
91.4
82.8
T13
90.0
72.9
47.7
92.1
88.9
91.2
83.4
T14
89.5
72.6
92.4
89.0
83.5
Toz-T14
90.2
71.8
47.2
92.2
89.1
AvgKD (Hinton et al., 2015)
Two
89.9
72.9
48.4
92.2
89.0
91.2
83.4
Toi-T14
83.7
67.9
29.4
88.0
83.2
84.6
73.6
TAKD (Mirzadeh et al., 2020)
Ti--T14
89.3
71.8
47.8
92.7
88.7
91.4
83.4
AutoSKDBERT (Ours)
T11-T14
92.0
76.9
51.8
93.0
89.3
84.3
D ORTHOGONAL EXPERIMENT OF AUTOSKDBERT WITH TRANSFORMER LAYER DISTILLATION AND DATA AUGMENTATION
This paper proposes a general distillation paradigm for BERT compression. Consequently, most of other distillation approaches combine with the proposed distillation paradigm. For instance, transformer layer distillation used in TinyBERT (Jiao et al., 2020) and MOEBERT (Zuo et al., 2022) be replaced with the stochastic KD paradigm proposed in this paper. Moreover, each teacher in the teacher team should has same hidden size with the student when distilling the transformer layer. Consequently, can distill the student with the teacher we implement list of orthogonal experiments to examine the effectiveness of the combination of AutoSKDBERT and TinyBERT, and show the experimental result Table 16.
used in this paper. In this section,
Table 16: Results of AutoSKDBERT with Data Augmentation (DA) transformer layer Distillation (TD) GLUUF-dev.
MRPC TT
RTE
CoLA
SST-2
QNLI
Model
DA
TD
Avg
Mcc
X
X
92.0
76.9
51.8
93.0
91.6
81.1
AutOSKDBERT
V
X
89.7
73.3
56.8
92.9
92.0
80.9
87.8
70.0
41.1
92.2
91.1
76.4
91.1
69.3
55.3
92.7
92.0
80.1
Due to the difference of hidden size between the strongest teacher T14 and the student, similar to TinyBERT, we employ also BERT BASE, i.e., T12, as the teacher for transformer layer distillation. TinyBERT employs random search choose the best batch size and learning rate from {16, 32) and (le-5, 2e-5, 3e-5), respectively. Differently, AutoSKDBERT uses also the categorical distributions on vanilla datasets with the batch size and the learning rate shown Table for each downstream task. Moreover, the epoch number for each downstream task can be found in Table 13.
As shown in Table 16, the combination of AutoSKDBERT and DA shows better performance com- pared to vanilla AutoSKDBERT on CoLA and QNLI tasks. Furthermore, the combination of Au- toSKDBERT. TD and DA achieves better performance compared to vanilla AutoSKDBERT on QNLI tasks. The main reason in consideration is that the categorical distributions are learned on the vanilla dataset instead the augmentation data. In the future, we will directly learn the categorical distribution the augmentation data.
18

---

Under review as a conference paper at ICLR 2023
However, the combination of AuROSK and TD is prone to obtaining worse performance each downstream task. We consider that the main cause of the above phenomenon is the knowledge transfer gap between transformer layer distillation and prediction layer distillation, i..., only using T12 for transformer layer distillation, Toi to 114 for prediction layer distillation. In the future, we will select appropriate teacher team to distill the transformer layer of BERT.
IMPACT OF HYPER-PARAMETHRR FOR CATEGORICAL DISTRIBUTION OPTIMIZATION
As ahove mentioned, there are important hyper-parameters, i.e., ineffective teacher number m and learning rate, for categorical distribution optimization. In this section, we discuss the impact of the above
hyper-parameters for the best performance of the learned categorical distributions. On the tasks of MRPC, RTE and CoLA, we implement AutoSKDBERT with m from to 10 and learning rate from 3e-4 le-3 with an interval of 1e-4, and show the results in Table 17.
Table 17: Results of ADBERT with various syper-parameters for categorical distribution optimization.
Learning Rate
Task Metric m
Std
3c-4
4c-4
5c-4
6c-4
7c-4
8c-4
9c-4
Ie-3
90.3
90.6
91.4
90.8
91.1
91.0
92.0
91.6
91.1110.6
91.6
91.6
91.8
91.6
91.5
91.7110.1
3
90.5
90.9
91.1
91.1
90.9
91.2
90.6
90.8
90.9:t02
4
90.3
90.1
90.6
91.0
91.3
90.4
91.0
90.4
90.6+t.4
MRPC Fl+aac
5
90.7
91.1
90.7
90.7
91.8
90.3
90.8
90.4
90.8
90.6
90.2
90.8
89.9
90.5+0.3
90.7
90.2
91.1
90.6
90.1
90.4
89.8
90.5
90.4++0.4
8
90.8
90.7
90.2
90.5
90.5
90.6
90.5
90.3
90.5+002
9 90.4
90.1
90.9
89.9
90.1
90.8
90.6
90.3
90.4+00.3
10
90.4
89.8
89.8
90.4
89.9
91.1
90.7
90.4
90.310.4
Mean-tStd
90.6
90.6
60.550.55
90.
0.5
90.
0.5
90.
0.6
90.8+
90.
7--0.6
6
90.
00.5
1 73.3
72.6
73.7
72.6
72.6
72.6
72.2
73.3
72.910.5
2 72.9
72.9
73.7
72.9
73.7
73.3
74.7
75.5
73.7+00.9
3
72.9
76.9
73.7
73.7
72.9
74.4
74.4
74.0
7..1
4 72.6
72.9
74.0
73.3
76.2
73.7
75.1
73.3
73.9
5
74.4
74.7
74.0
73.3
74.7
74.0
74.7
75.5
74.4.0066
RTE ace
6
75.5
76.2
74.0
74.7
75.5
75.1
76.2
76.2
75.410..7
7
74.1
75.1
72.2
74.4
74.4
74.0
74.0
74.4
74.1
8
75.8
73.7
74.4
74.4
74.0
74.4
74.5+0.6
9 74.0
73.3
73.7
73.7
74.4
74.4
74.4
76.9
7,.4+10
10
72.9
74.7
72.6
75.1
73.7
73.7
74.7
74.0
73,9
Mcan- Std
73.8
74.3
1.4
73.6
73.8
74.2-+1.0
74.0
74.5
74.8
50.0
50.0
50.0
49.7
50.6
49.6
50.0
2
50.0
50.9
49.8
50.3
50.2
49.8
49.2
50.2
50.0+-0.5
3
48.3
48.6
49.4
49.8
50.6
50.2
50.1
49.9
49.6+-0.7
4
49.9
49.3
49.8
50.8
51.8
50.5
49.4
51.5
50.4+10.
49.0
50.0
49.7
49.3
49.4
49.410..
CoLA Mec
6
48.7
49.3
50.0
49.3
49.5
49.7
49.8
49.4
49.5100.4
7
49.3
48.7
50.0
49.3
49.7
49.7
49.2
49.0
49.4+0.4
8
49.6
49.3
49.1
49.4
49.9
50.5
50.6
50.2
49.8+
9
49.8
49.7
50.1
49.0
49.2
48.7
50.2
49.6
49,5+0.5
10
49.2
49.6
48.5
50.6
49.7
49.4
49.410066
Mean=SSt
69
49,7
49.60
50.1
49.8
49.8 49,7+0.8
We can draw a conclusion from Table 17 that the proposed AutoSKDBERT is sensitive to inef- fective teacher number rather than the learning rate. For each task, the ineffective teacher number m plays more important role compared to the learning rate for AutoSKDBERT. For instance, the mean value of m 2 is 91.7 which is 1.4 higher than the mean value of the largest difference of mean value with respect to learning rate is only 0.2. There similar phenomenon the tasks of RTE and CoLA. Therefore, the above conclusion be drawn.
10. However,
19

---

Under review as a conference paper at ICLR 2023
F COST COMPARISON OF TINYBERT AND AUTOSKDBERT
In this section, we show the cost of AutoSKDBERT in terms of categorical distribution optimization and evaluation, and compare our approach to TinyBERT with respect to algorithm cost. Experi- mental results are shown in Table 18 where on five downstream tasks, the cost of AutoSKDBERT is 38.72 hours which is 8.4x less than TinyBERT. Moreover, we obtain the cost on NVIDIA A100 GPU with AMD EPYC 7642 48-Core Processor.
Table 18:
The cost (hours) comparison of TinyBERT and AutoSKDBERT on five downstream tasks. These results about TinyBERT are obtained by following the experimental settings deseribed in Jiao et al. (2020) with the code publicly released by the authors at https: /ttttss://giii h/tp//gittttt com/ huawei wwwaeiooooh/araaaaaeeeeeeee Language-Menn Modee 1/t /treee/e///eeeeeeeeeeeeeeeeeeeee wwwssttrrriiyyBRR...
TinyBER'
AUtOSKDBERT
MRPC RTE
CoLA SST-2
QNLI
MRPC
RTE CoLA SST-2 QNLI
Transformer Layer Distillation
3.42
2.80
12.72
12.90
61.94
0
0
Categorical Distribution Optimization
0
0.26
0.17
1.06
1.28
2.45
Prediction Layer Distillationf
5.04
3.24
4.35
31.71
187.62
1.75
1.25
5.75
6.50
18.25
Total Cost
8,46
6.04
17.07
44.61
249.56
2.01
1.42
6.81
7.78
20.70
For TinyBERT, the cost is obtained by groups of experiment with various hyper-parameterr (i..., batch sizes of (16, 32) and learning rates (1o-5, 20-5, 36-5)) on augmentation data. For AuoSKDBERT, cost obtained by groups of experiment on vanilla, data with different categorical distributions learned in the process of categorical distribution opimizaaiinn.
The distillation process of TinyBERT can be divided into two phases: 1) transformer layer distilla- tion augmentation data and 2) prediction layer distillation on augmentation data. The transformer layer distillation of TinyBER' time-consuming, e.g., it spends about 62 hours on QNLI. Besides, the prediction layer distillation of TinyBERT is also time-consuming due to using large-scale aug- mentation data.
Differently, AutoSKDBERT consists of categorical distribution optimization and evaluation (i.e., prediction layer distillation). On the one hand, categorical distribution optimization is efficient, e.g., 2.45 hours the task of QNLI, due to the gradient-based SSWO. On the other hand, categori- cal distribution evaluation is also efficient even choosing the best categorical distribution from 25 candidates.
G TINYBERT WITH STRONGER TEACHER MODEL
AutoSKDBERT employs two stronger teacher models, i.e., Tia and T14, compared to most of the comparative methods shown in Table To verify the impact of strong teacher on the distillation performance of other paradigms, we employ T12 and T14 as the teachers to distill TinyBERT on five downstream tasks for fair comparison. Following TiRyBERT (Jiao al., 2020), we implement the experiments with batch sizes of (16, 32} and learning rates of (le-5, 2e-5, 3e-5), and choose the best result to show Table 19.
Table 19: Results of TinyBERT with the strongest teacher Tıa on GLUE-dev. These re- sults are obtained by TinyBERT with the fine-tuned teacher model of AutoSKDBERT using the code publicly released by the authors at https: // wtw. con/huawei-nod Pretrai - lLanguge Mode /tttt//eeeeteeeeeeeeeee/eeeeeeet /tree//mstterr/hyyaa.r
MRPC
RTE
CoLA
SST-2
QNLI
Student
Teacher
Avg
Mec
acc
T12
87.0
67.9
42.2
92.0
91.2
76.1
TinyBERT (w/o aug)
T14
86.7
70,0
40,9
92.0
90.9
76.1
We can observe that the strong teacher T14 contributes to only improving the performance on RTE. For the above phenomenon, the main reason is that a capacity gap (Mirzadeh et al., 2020) exists between T14 and student which is prone obtaining unsatisfactory performance. As result, conclusion can be drawn that the stronger teacher T can not always contribute to improving the performance of other distillation paradigms.
20

---

Under review as a conference paper at ICLR 2023
H MT-BERT FOR BERT COMPRESSION
For BERT-style language model compression, we verify the performance of MT-BERT (Wu et al., 2021) whose objection function can be expressed as:
CE 1 CF.
LITBERT
where, N indicates the number of teachers, CE(...) :) is the cross-entropy loss, denotes the tem- perature, y represents the ground-truth label, Yi and ys refer the outputs of i-th teacher and the student, respectively.
We employ Tio T14 the teacher team distill the student via Eq. 9. Particularly, only use the weighted multi-teacher distillation loss without the multi-teacher hidden loss and the task- specific loss as MT-BERT (Wu al., 2021).
The hyper-parameters are given as follows:
Learning Rate: {le-5, 2e-5, 3e-5} for all tasks.
Batch Size: {16, 32, 64).
Epoch: 10 for MRPC, RTE and CoLA tasks, for other tasks.
Other settings follow AutoSKDBERT.
I DETAILS GLUE BENCHMARK
GLUE consists of NLP tasks: Microsoft Research Paraphrase Corpus (MRPC) (Dolan & Brock- ett, 2005), Recognizing Textual Entailment (RTE) (Bentivogli et al., 2009), Corpus of Linguis- tic Acceptability (CoLA) (Warstadt al., 2019), Semantic Textual Similarity Benchmark (STS-B) (Cer et al., 2017), Stanford Sentiment Treebank (SST-2) (Socher et al., 2013), Quora Question Pairs (QQP) (Chen al., 2018), Question NIJI (QNLI) (Rajpurkar al., 2016), Multi-Genre NLI (MNLI) (Williams et al., 2017), and Winograd NLI (WNLI) (Levesque et al., 2012).
MRPC belongs to a sentence similarity task where system aims to identify the para- phrase/semantic equivalence relationship between two sentences.
RTE belongs to a natural language inference task where system aims to recognize the entailment relationship of given two text fragments.
CoLA belongs to single-sentence task where system aims to predict the grammatical correctness of an English sentence.
STS-B belongs sentence similarity task where system aims to evaluate the similarity of two pieces of texts by score from to
SST-2 belongs to single-sentence task where system aims to predict the sentiment of movie reviews.
QQP belongs to a sentence similarity task where system aims to identify the semantical equiva- lence of two questions from the website Quora.
QNLI belongs to natural language inference task where system aims to recognize that for a given pair <question, context>, the answer the question whether contains in the context.
MNLI
belongs to natural language inference task where system aims to predict the possible relationships (i.c., entailment, contradiction and neutral) hypothesis w.r.t. premise for a given pair <premise, hypothesis>.
21

---

Under review as a conference paper at ICLR 2023
WNLI belongs to natural language inference task where system aims to determine the referent of sentence's ' pronoun from list of choices.
22


Reviews:
[{"review_id": "c025OJo6q_", "reviewer": "Reviewer_n9fn", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes a knowledge distillation approach with multiple teachers by introduce a weighting on teachers, and the weights are optimized in a bi-level manner. It achieves state-of-the-art results on several GLUE tasks compared with other model compression methods.", "summary_of_the_review": "This paper proposes a reasonable approach to weight multiple teacher in knowledge distillation, and gets good results on the GLUE benchmark, but the results are not discussed and analyzed thoroughly, and the paper writing is comparatively weak.", "strength_and_weaknesses": "Strength:\n* The authors propose a reasonable bi-level optimization for the weighting of teachers and achieve strong results.\n\nWeakness:\n* The paper writing is comparatively weak, e.g., The figure 1 is even harder to understand than the main text part.\n* Lack of meaningful analysis and discussion towards the result numbers. For example, overall the proposed approach performs on par with MoBERT. Why in some tasks it performs worse? Is the proposed approach orthogonal to the previous methods? Will the performance be better if multiple techniques are applied together?", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works.", "empirical_novelty_and_significance": "Not applicable"}, "extra_text": {}}, {"review_id": "b9Vtmuom_go", "reviewer": "Reviewer_SQHt", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "Given a predefined teacher team, the paper aims at finding an optimal categorical distribution of these teacher models for student network distillation. Rather than simple ensemble learning (uniform weight), the paper proposes a two-stage selection and optimization strategy. In the first stage, ineffective teachers are identified and discarded by optimizing the categorical distribution. In the second stage, the categorical distribution is further tuned for effective teacher teams. The paper conducts experiments on the GLUE benchmark and make comparison with the most relevant works such as DistilBERT and TinyBERT.", "summary_of_the_review": "Although the proposed method achieves decent results, the technical novelty is quite limited. Optimizing the categorical choice while optimizing the network parameters is a widely studied bilevel optimization problem in neural architecture search and other areas. Another important concern is that a more simple baseline should be considered such as single best teacher and Gumbel softmax optimization to systematically study whether it is necessary to optimize the categorical distribution.", "strength_and_weaknesses": "**Strength**:\n\n1. The formulation of the problem is clear and the paper is well-written.\n2. Most relevant works are discussed. The experiments show that AutoSKDBERT achieves good results.\n\n**Weaknesses**:\n1. To show that optimizing the categorical distribution is necessary, I wonder whether a single-teacher KD is enough. Table 1 shows some results. However, I would like to see the complete results from T1 to T14. Besides, does \"Best Single Teacher\" refers to the teacher that can achieve the best performance for student distillation or the teacher itself that achieves the best performance?\n2. The novelty of the proposed method is limited. ", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "KiSI1cU-Km", "reviewer": "Reviewer_jsZh", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper proposed a new knowledge distillation paradigm for BERT, AutoSKDBERT, which stochastically sampled a teacher from the predefined teacher team by a categorical distribution. They proposed a two-phase optimization framework to better incorporate the categorical distribution and student model. To alleviate the gap between categorical distribution optimization and evaluation, they proposed an SSWO strategy for optimization. The experimental results show superiority compared with strong baselines.", "summary_of_the_review": "Overall, the proposed method is simple and easy to implement. And it could raise better performance than SOTA. But there are lots of tricks to boost the performance, like 1. generating 25 candidates, training 25 student models, and choosing the best one; 2. mutual-tuning $m$; 3. using a stronger teacher model (*whole word masking*). However, there lack of details and analysis of the tricks. Further, the clarification should be clearer.", "strength_and_weaknesses": "### Pros:\n1. The proposed method is simple yet effective, and it provided a more flexible way to give full play to the teacher team.\n2. The experimental results show obvious superiority compared to the SOTA baselines on the GLUE benchmark.\n\n### Cons:\n1. One critical problem is how to choose the ineffective teacher number, i.e., $m$. It is a very important parameter in the whole paradigm. However, as claimed in Sec 3.2.2, $m$ is a hyperparameter and it may differ a lot for different tasks.\n2. As claimed in Sec 3.3.2, AutoSKDBERT delivers 25 categorical distribution candidates and selects the best one. How to produce 25 candidates in detail?\n3. As stated in Sec 3.3.1 and Sec 4.1, are there 25 categorical distribution candidates in both phase-1 and phase-2? Or phase-1 will only produce 25 candidates in the ablation study experiments?\n4. In Sec 3.3.1, what does *trains them from scratch* mean? What is *them*? the student model?\n3. In figure 2, we can see $T_{14}$ plays a significant role, but $T_{13}$, which has the same number of parameters as $T_{14}$, is much weaker. The only difference is that $T_{14}$ adopts *whole word masking*. Since *whole word masking* is signally effective, why not adopt it on other teacher models? \n4. Because $T_{14}$ is a new type of BERT model with better performance, do the baselines use such a model as one of the teacher models? If not, what are the results without $T_{14}$? What if the baselines use the same teacher models?\n6. Since some teacher models, e.g., $T_{14}$, perform distinctly better, it will be good to compare with the best single-teacher KD.\n5. What's the number of teacher models of the baselines? Is it a fair comparison if the baselines use fewer teacher models than 14?\n8. In Sec 4.2, for the CR setting, the training involves all the teachers, why not use the same way in evaluation? Then it will be consistent between training and evaluation and can compare with SSWO more fairly.\n1. It seems unclear about the categorical distribution initialization module. Why is the initialized $\\theta_i$ like shown in Figure 1, i.e., why are 0.199, 0.201, 0.199, 0.200, and 0.201?\n\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "Not applicable"}, "extra_text": {}}, {"review_id": "-OEyBHZvPq", "reviewer": "Reviewer_FHc1", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "In this paper, authors proposed AutoSKDBERT, a new knowledge distillation paradigm for BERT compression, that stochastically samples a teacher from a predefined teacher team following a categorical distribution in each step, to transfer knowledge into student.\n\nWith extensive experiments on GLUE benchmark show that the proposed AutoSKDBERT achieves state-of-the-art score compared to previous compression approaches on several downstream tasks.", "summary_of_the_review": "I choose the final decision 'marginally below' given my questions above as well as the proposed method for multi-teacher is only studied on BERT. If authors are able to clarify and extend the work, I would be happy to reconsider the decision.", "strength_and_weaknesses": "Strength \nAuthors proposed a novel learning algorithm to learn from multiple teachers.\n\nWeaknesses\nThis paper only focused on bert. My understanding is the proposed method can be adapted to other network architecture. The impact would be larger if such method is independent of the network architecture of teacher/student. ", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject