Co-training and Co-distillation for Quality Improvement and Compression of Language Models
Authors: Hayeon Lee
Rui Hou
Jongpil Kim
Davis Liang
Hongbo Zhang
Sung Ju Hwang
Alexander Min
Venue: EMNLP 2023 Findings (2023)
Keywords: Knowledge Distillation
TL;DR: Co-Training and Co-Distillation (CTCD) framework that improves the quality of PLMs while increasing their inference efficiency.

Abstract: Knowledge Distillation (KD) compresses computationally expensive pre-trained language models (PLMs) by transferring their knowledge to smaller models, allowing their use in resource-constrained or real-time settings. However, most smaller models fail to surpass the performance of the original larger model, resulting in sacrificing performance to improve inference speed. To address this issue, we propose Co-Training and Co-Distillation (CTCD), a novel framework that improves performance and inference speed together by co-training two models while mutually distilling knowledge. The CTCD framework successfully achieves this based on two significant findings: 1) Distilling knowledge from the smaller model to the larger model during co-training improves the performance of the larger model. 2) The enhanced performance of the larger model further boosts the performance of the smaller model. The CTCD framework shows promise as it can be combined with existing techniques like architecture design or data augmentation, replacing one-way KD methods, to achieve further performance improvement. Extensive ablation studies demonstrate the effectiveness of CTCD, and the small model distilled by CTCD outperforms the original larger model by a significant margin of 1.66 on the GLUE benchmark.

Co-training and Co-distillation for Quality Improvement and Compression of Language Models
Hayeon Lee' Rui Hou' Jongpil Kim
Davis Liang
Hongbo Zhang' Sung Ju Hwang Alexander Min Meta Ar" Abridge Al KAIST
hayeonlee@meta..c com rayhouêmeta.c com jpkimêmeta.com
daviseabridge. com hbzhangêmeta.co com sjhwang878kaistt.cc ac. kr alexminêmeta. com
Abstract
The ideal scenario would involve compressing the teacher match the size of the small student without any performance drop. However, despite intense research in the area of KD (Ture et al., 2019; Tsai et al., 2019; Tang et al., 2019; Jiao et al., 2020; Sanh et al., 2019a; Sun et al., 2019; Wang et al.. 2020b,a), none of the existing methods have successfully avoided performance degradation dur- ing the distillation process. Some approaches have attempted to mitigate the performance gap by in- corporating external factors. For instance, Jiao et al. (2020) incorporates data augmentation, while Wang et al. (2020b,a) focuses on designing student architecture. However, these approaches have lim- itations as they only provide supplementary tech- niques rather than addressing the fundamental issue of performance loss in KD. This raises an interest- ing question: Can we compress model without scarifying the performance through KD?
Knowledge Distillation (KD) compresses com- putationally expensive pre-trained language models (PLMs) by transferring their knowl- edge to smaller models, allowing their use in resource conburcccc or real-time settings. However, most smaller models fail to surpass the performance of the original larger model, resulting in sacrificing performance improve inference speed. To address this issue, we pro- pose Co- Co-Training and Co-Distillation (CTCD), a novel framework that improves performance and inference speed together by co- co-training two models while mutually distilling knowledge. The CTCD framework successfully achieves this based on two significant findings: 1) Distill- ing knowledge from the smaller model to the larger model during co-training improves the performance of the larger model. 2) The en- hanced performance of the larger model further boosts the performance the smaller model. The CTCD framework shows promise it can be combined with existing techniques like ar- chitecture design or data augmentation, replac- ing one- one-way KD methods, achieve further performance improvement. Extensive ablation studies demonstrate the effectiveness of CTCD, and the small model distilled by CTCD outper- forms the original larger model by significant margin of 1,66 the GLUE benchmark.
In this work, propose novel framework called Co-Training and Co-Distillation (CTCD) for improving the performance of language model while compressing it through KD. CTCD involves jointly training the teacher and the student models, allowing them to transfer knowledge to each other bidirectionally, from the teacher to the student and vice versa. Our work uncovers two key findings within the CTCD framework: Firstly, we demon- strate that transferring knowledge from the smaller model to the larger model during co- co-training sig- nificantly improves the performance of the larger model. other words, by employing knowledge distillation (KD), we enhance the performance of the teacher model compared to standalone training. This is significantly different from conventional one-way KD, where the teacher model cannot ben- efit from the distillation process since is longer trained or distilled. Secondly, the improved perfor- mance of the larger model leads to further enhance- ments in the performance of the smaller model through KD from the teacher model to the student
1 Introduction
In recent years, the high computational cost of pre- trained language models (PLMs) (Radford et al., 2019; Yang et al., 2019; Dai et al., 2019; Shoeybi et al., 2019; Li et al., 2020; Brown et al., 2020) become constraint in serving resource-limited real-time applications. Knowledge Distillation (KD) (Hinton al., 2015; Romero al., 2015) is a popular model compression technique to tackle this issuc, with a smaller model (student) lcarning (distilling) from a larger model (teacher).
Work done while interning at Mcta AI.

---

Text Corpus
tesk-ognostic Knowledgee Distilaatonn
Arinttated Dota
Pretra ned Languuage
Fine tuteet
Mode!
Prefrsitingg
adaptaation rinndtninn
Tank
Figure I: Task-agnostic KD method The proposed framework are task-agnostic methods that perform KD during the pre pre-tranning stage. This allows the distilled model to be deployed and fine-tuned on various downstream tasks.
model. These novel findings enable the smaller model to surpass the independently trained larger model while maintaining its inference efficiency within the CTCD framework. Moreover, the CTCD framework be combined orthogonally with isting techniques for external factors such stu- dent architecture design Wang al. (2020b,a) or data augmentation (Jiao al., 2020). We achieve this by replacing traditional one-way KD methods with the proposed CTCD framework, which holds promise for significant performance improvements in language modeling tasks.
of the larger model further enhances the perfor- mance of the smaller model. In our experiments, the student model compressed by CTCD framework obtained 1. 1.66 higher gain than the original large model trained using stand-alon. method, demon- strating that our approach can improve model qual- ity and inference efficiency concurrently. In summary, our contributions are as follows:
We propose a novel knowledge distilla- tion framework called Co-Training and Co- Distillation (CTCD) framework to improve the performance of models while compressing them through KD.
In addition our CTCD framework, introduce Community KD that utilizes CTCD framework with the pre-trained teacher model by combining CTCD with conventional KD. Community KD dis- tills knowledge from two students to each other, as well as from the pre-trained teacher to each student, during the co-training of the students. We show that distilling from the other student as well as the pre-trained teacher is better than only may dis- tillation from the pre-trained teacher, is done in conventional KD. Note that the inference cost on a task the same as in conventional KD, as we take one of the students distilled by Community KD when deploying it to the task.
Through our experiments, we demonstrate that distilling knowledge from the smaller model to the larger model during co-training improves the performance the larger model.
Additionally, we highlight that the enhanced performance of the larger model further boosts the performance of the smaller model, result- ing in improved performance for both models.
We provide valuable insights into adjusting loss weights and the length of the training phase for the effective application of the CTCD framework through extensive ablation study.
We validate the effectiveness and efficiency of CTCD on the GLUE benchmark (Wang al., 2019), which contains different language under- standing tasks. As shown in Figure/ 1, We focus on task-agnostic KD scenarios, where KD occurs during the pre-training stage. Then, we fine-tune the distilled PLMs on each downstream task and evaluate their performance. This is more challeng- ing compared to task-specific KD, as it requires substantial computational resourees to train PLMs a large text corpus, and the distilled knowledge should be transferable across diverse downstream tasks. Our extensive ablation study revealed that the larger model benefits from the distillation of the smaller model, and the performance improvement
2 Related Work
One-way Knowledge Distillation Knowledge distillation (KD) (Hinton et al., 2015) is a model compression technique in which a smaller student model distills knowledge from a larger, pre-trained teacher model. Recently, many rescarchers have explored KD address a high computational com- plexity of language model resulting from its in- creasing size. This includes task-agnostic KD meth- ods for the pre- pre-training stage (Sanh et al., 2019a; Wang et al., 2020b,a), task-specific KD method for

---

propagation in Loss Tearcher hard loss teacher soitToss
Co-distillation
Teacher
Co-training
Teacher
Large Text Corpus
Saft Label
teacher hard loss
Prediction
studen soft los
seecher sof Boss
Scratch
Bw
BWW
Hard Label
Stodant baid
Student
||-I0
Prediction
Student Soft Label
Scratch (U WV
Co-training
Back propagation
Figure 2: Co-Training and Co-Distillation (CTCD) During training, the teacher and student models can learn more effectively by comparing their prediction outputs not only against the ground truth but also against each other's predictions. We refer to the former a "hard" loss and the latter a "soft" loss. For example, the student soft loss captures the distance between the student's prediction and the teacher's 's prediction, and vice versa. This co-training and co- distillation approach improve the performance of the teacher model, which in turn benefits the performance of the student model.
the fine-tuning stage (Sun et al., 2019), and both of pre-training and fine-tuning stages (Jiao al., 2020). Additionally, Wang al. (2020b,a) have redesigned the architectures of student language models. However, the ony KD can lead a loss knowledge, resulting in smaller models that generally have difficulty matching the performance of larger models, leading to performance degrada- tion. In contrast, our CTCD can improve the quality of both teacher and student models, making it pos- sible for the student achieve the same quality as the original teacher model.
Co-training trains two (different-sized) models (e.g., a teacher and student) concurrently with the goal of achieving similar model quality.
Co-distillation transfers knowledge in both di- rections between two models (e.g., a teacher and student), during co-training.
Figure 2 illustrates how co-trained models learn together by comparing their prediction outputs against predictions from each other and the hard labels (or "ground truth"). We refer to the former as a "soft" loss and the latter as a "hard" loss. For instance, the soft loss of the student model mea- sures the accuracy of the student's 's prediction by considering the teacher's prediction as soft label, and vice versa.
Reversed Knowledge Distillation Recently, re- searchers (Yuan et al., 2020; Qin al., 2022; Lee et al., 2023) have demonstrated that reversed Knowledge Distillation (reversed KD), which trans- fers knowledge from smaller or poorer model to a larger model, can improve the performance of the student model. In particular, Qin et al. (2022); Lee et al. (2023) investigated the upplication of reversed KD in the PLMs. showing that larger model can benefit from a poorer and pre-trained model for a specific downstream task. Inspired by the suc- cess of reversed KD, we design a co-distillation framework that includes reversed KD to improve the performance of the teacher model by distilling knowledge from the smaller student model. Unlike existing reversed KD methods, which are limited improving the performance of the larger model, our proposed co-distilaation framework can achieve both performance improvement and model com- pression, by showing a better-quality teacher leads to a better-quality student.
Task Formulation classification task with K classes. For each training instance x and its ground truth label y, we denote that the ground truth distribution over the labels is (kka.r) (g(k) for simplicity) where for each label l {1...K), q(v) = 1 and ((k) 0 for all A A y. For each x, the teacher model tæp parameterized by & and the student model se parameterized by 0 predict the probability of each label & as D (kl]z) and p (k) respectively as follows:
Suppose that we are given a
pb(k))) f(z') exp(xt/T)
ΣK exp(21/T)
expl=1/7)
pb(k)2)) = f(2²) = ΣK exp(2 /T)
where is the softmax function, 2' {=]}<< (x(x) is the output logit of the teacher model. z (2]] ] so (x: is the output logit of the student model, and 5 is the temperature to soften (A: and (k)).
3 Co-training and Co-distillation
We first introduce the concepts of co-training and co-distillation briefly:

---

Teacher
Teacher
Pre-trained
Label
Large That Corpus
Co-distillation
Protittion
Student
Scratch
Book propagation
Soft Label
000
Loss hard loss seft loss from student soft loss from teacher
Hard Label
Student
Co-training
Tredition
Student
Scratch
propapation
Soft Label
Co-training
Figure 3: Community KD Different from conventional KD, where each student learns from the prediction of the pre-trained teacher only, the proposed approach learms each student from hoth prediction of the pre-trained teacher and prediction of another student during co-training. Note that since we take one of the pre-trained students to adapt it to downstream tasks, the inference cost is the same as the student training with the conventional KD.
The proposed objective (CTCD(O, (0,0 Φ) consists of normal KD objective LKDZ >s to distill knowl- edge from the teacher model to the student model and reversed KD objective D: distill knowledge from the student model the teacher model, during their co-training.
where StopG( denotes that we do not compute the gradient of z. In the conventional KD method, Equation (2) is the final objective to learn the stu- dent model only with the pre-trained teacher model.
Student Teacher Different from such a one- way KD method, we introduce the reversed KD objective LReKD:-=((topGi Φ) to train the teacher model ls as follows:
The normal KD objective LKD4 (f, <)) aims to train the student model by minimizing weighted sum of the cross-entropy loss (g. po) between the ground truth g and student prediction pe and Kullback-Leibl diver- gence (KI. divergence) DPp" pb) between the pre- dictions of the student and the teacher as follows:
Teacher - Student
LRcKDD x (StopG Φ)
(3)
Bn. (1(g. + B. ((pg.p5)
where Bn. and B, are weighting hyper-parametar values of the hard loss H PΦ) and soft loss D(p p²)) for the teacher model, respectively. By minimizing KL divergence D(p pb) between the predictions of the student model (p%) and the teacher model (p)), the teacher model learns from the student model. In the reversed KD objective, we only train the teacher model by applying (topG(z) to the gradient of the student parameters 6.
LKD(0, an H(q.Po) as P
where
(g:Po)) = Σ (k)) log(po(k)),
1=1
p (k) Σ P (k) log
D(P).P))
P (k))
Co-training With the Equations (2) and (3), we get the final objective LeTCn () as follows:
ag and are weighting hyper-paramete val- ues for the cross-entropy loss and KL divergence. respectively. We regard the cross-entropy loss H (() the hard loss for the student model, KL divergence (P Pg) as the soft loss for the student model, and following BERT (Devlin et al., 2019), H (e) denotes the Masked Language Modeling loss (MLM loss). the KD objective, we consider the teacher parameters & as constant since we only train the student parameters 0 while the teacher model lg is fixed:
0- oh argmin CoTcD(O.
0.6
LKD(O, StopG(())))) + LReeD ())
(4)
Adapting to Downstream Task After model co- training/-distillttion, the trained smaller (student) model sg- with trained parameter 0" can be de- ployed for multiple downstream tasks improve inference efficiency. To fine-tune the model for a specific downstream task, we adapt the trained parameter 0" using the dataset for that task.
LK (0 (to))())))))
(2)

---

4 Community KD
80
79.94
Furthermore, we introduce an advanced CTCD appli- cation named Community KD that can utilize CTCD framework with the pre-trained teacher model, as shown in Figure 3. It consists of pre-trained teacher for with pre-trained parameters Ø and two students s0, and s0, parameterized by 0ı and 02, respectively. During the co-training of two stu- dents, each student learns from the hard labels, soft labels generated from the pre-trained teacher pre- dictions, and soft labels generated from other stu- dent predictions. In other words, we conduct one- way knowledge distillation from the pre-trained teacher to each student by minimizing KL diver- gence between the teacher prediction and predic- tions of each student ( P(.)) and 2 Po.. and co-distillation between students in both direc- tions by minimizing LCTED 01, 02). The final objective (01, 02. StopG((")) is as follows:
Average Performance
79
1.88
78.06
78.08
0.47
77.94
0.88
T7.48
77.06
77
76
Stand-alon
Co-training & one-way distilation
Co-trainig co-distilation (Curs)
75
Teacher (67M)
Student (2.
Figure 4: Average performance on dev sets of GLUE benchmark after training epochs Learning from the stu- dent improves the performance of teacher by an average of 1.88. Such improvement in teacher performance leads to improvement in student quality from 77.46 to 77.94.
for 20 epochs in Section 5.1. To analyze the ef- fectiveness of CTCD method, we train the teacher and student models for 10 epochs and 20 epochs in Section 5.2 and Section 5.3, respectively. In Section 5.4, we train the models for 3 epochs after parameter remapping, which is the same as in ventional one-way KD method (Sanh et al.. 2019a).
07, 0 argmin (Ca, 02 StopGi("")
01.02
(0 :(pr, D(poy: Po-))
(5)
Training Time For the one-way distillation, we need GPU day train the teacher for 10 epochs and 1.3 GPU days to distill knowledge from the pre-trained teacher to the student for another 10 epochs, which consumes a total of 2.3 days. For CTCD, it takes 3 GPU days to train both teacher and student models from scratch for 10 epochs. We use the automatic mixed precision (AMP) of Py Torch (Paszke et al., 2019) to accelerate training for all our models.
We select one of the two students distilled by Community KD (0* 0f 0" 02) and fine- tune the selected single student sø- for downstream tasks, resulting that the inference cost does not in- crease compared with the conventional KD method.
5 Experiment
We present a comprehensive analysis of the pro- posed CTCD method through empirical experiments. Section 5.1, we validate our CTCD method by comparing the performance of small models dis- tilled by CTCD to the original large model on the GLUE benchmark (Wang et al., 2019). In Sec- tion 5.2, we analyze the impact of co-distillation by adjusting loss weights for the soft losses of student and teacher. In Section 5.3, study the impact of training length CTCD method, allow- ing us determine the optimal training length for CTCD method. In Section 5.4, we demonstrate the efficacy of Community KD by comparing the per- formance of a model distilled by the Community KD to a model distilled by the one-way KD.
Dataset To validate our CTCD method, we use a reduced dataset (30M) created by uniformly sam- pling out of every 4 sentences from the original pre- pre-training dataset (BookCorpus (Zhu et al., 2015) + Wikipedia (Foundation)) used in the conventional me-wway KD method (Sanh et al., 2019 9a). We eval- uate our distilled models on the dev sets of the GLUE benchmark (Wang al., 2019), which con- sists of nine sentence-level classification tasks. Section 5.4, we use the original pre-trate mre-training dataset to train Community KD method.
Model Architecture We use 6-layer BERT (De- vlin al., 2019) model as the teacher and 4-layer BERT model as the student to analyze the effec- tiveness and efficiency of our CTCD method. In Sec- tion 5.4, we use pre-trained BERT-base model as the teacher and a 6-layer BERT model as the student.
Implementation details We use learning rate of 5c-4, linear warm-up of 5%, AdamW opti- mizer (Loshchilov and Hutter, 2019), and batch size 128 with 100 GPUs for pre-training We train the teacher and student models from seratch

---

Performance
GAP w/ Teacher
Original Teacher
78.06
One-way Distil. (10 epoch)
77.46
-0.60
CTCD (10 epoch)
77.94
-0.12
Student
One-way Distil. (20 epoch)
78.39
+0.33
CTCD (20 epoch)
79.12
+1.66
Table 1; Average performance on dev sets GLUE benchmark The student distilled by CTCD significantly outperforms the original teacher trained using the stand-alows aand-alone method, achieving a higher gain of 1.66.
2.0
1.0
80
"Suudin weighte band surach besmi:t
hort lonc soth cuan111
"Teacher wrights
0.86
0.88
.0
Perforancs CAP alend---ddaa woo--lanne
sudpe pal--paas yM dvS aounuoonee
0.74
0.8
Average Performnce
1.5
Average Performance
79
78
77.91
0.84
77.94
-.88
$7.79
78.42
1.0
77.69
78.06
77
78
0.5
06
75.05
77
stand-afone 1:1
1:2
stand-alone
1:2
1:4
4:4
weight for hard lose: weight for soft loss
weight for hard loss: weight for soft loss
(a) Teacher Performance
(b) Student Performance
Figure 5: Adjusting Loss Weight We investigate the impact of the distillation for the teacher model and student model by adjusting loss weights (an, Br. (B.) for hard loss and soft loss. (a) We (co- Jtrain the teacher model distilling knowledge from the student by fixing knowledge from the teacher by fixing 3h 8. : and varying оn on the large text corpus. Then report the average performance of each pre-trained model after fine-tuning downstream tasks (dev sets) of GLUE benchmark.
1: and varying 3, 89 the large text corpus. (b) We (co-)train the student model distilling
without using any knowledge distillation. Co- training & One-way distillation trains teacher and student models together from scratch, with knowledge only flowing from the teacher to the student. Co-training & Co-distillation (Ours) is CTCD method, which trains both teacher and student models together from scratch and distills knowl- edge between each other in both directions. For distillation methods, we set the weights of the hard losses for the teacher and student to 1. The weights of the soft losses are chosen from the set (0.5, 1, 2), and the results are reported with the best-performing weights.
46
44
5.22
Matthew Correlation
42
40.94
#1.01
-5.92
O 35.02
Pre-training Epoch
O Performance GAP
Ours (Co-traní-disil)
Conventional KD
Figure Length Training pre-trained student models under two different training lengths 10/20 epochs while dis- tilling knowledge from teacher models via ours or the conven- tional KD method. Then we adapt pre-trained student models CoLA task. With enough longer training (20 epoch), the student model distilled hy ours significantly outperforms the student model distilled by the conventional KD method, with a higher gain of 5.22.
1) Overall Results As shown in Table 1, the stu- dent distilled by CTCD method significantly out- performs the original teacher trained using the stand-alone method on the average performance of the GLUE benchmark, achieving higher gain of 1.66. Furthermore, the student model distilled by our CTCD method outperforms the student distilled by the one-way distillation method on the aver- age performance of the GLUE benchmark. After training for 10 and 20 epochs, the student distilled by our CTCD method consistently has higher gains
5.1
Could Knowledge Distillation Help Improve Performance?
Table and Figure 4, we show the average per- formance models trained using different methods on a large text corpus and fine-tuned against the GLUE benchmark. Stand-alone trains a model

---

than the student distilled by one-way distillation, as 77.46 vs. 77.94 and 78.39 vs. 79.12, respectively. 2) Does learning from a small and weak student provide a performance benefit to the teacher? As shown in Figure 4, the distillation of knowledge from the student model to the teacher model has been shown to significantly improve the quality of the teacher model, with average increase of 1.88 compared teacher training methods that do not incorporate such distillation process (such Stand- alone and Co-training & one-way distillation).
weight the soft loss of the teacher during training (aq, : as : 3/ 3. 2->1:1: 4) leads to improved performance, with an average score of 78.42 - 78.83 - 79.86. Furthermore, we observe that emphasizing only the soft loss of the teacher (1 : 1: 1 4) yields better performance than emphasizing both the hard and soft losses of the teacher (1 1 4 4), with average score of 79.86 vs. 79.09.
Student side We find that the student model's performance is not sensitive to the weighting values for the hard and soft losses of the student, (an, ag). Regardless of the chosen values, co-training the student with distillation consistently improves its performance compared training the student alone. For instance, when we emphasize the soft loss of the student by increasing the weighting value for (ox)) as 1 1 1 : 2 1 1 1 4 1 1, we observe similar levels of performance for the student model.
3) Is the teacher's performance improvement reflected in the student's performance improve- ment? Furthermore, we find that a better teacher leads to further performance improvement of the student from 77.46 to 77.94. The results demon- strate that the distillation process successfully im- proves the performance of both the teacher and student. The student trained with our CTCD method achieves better performance than students trained with the Stand-alone or Co- Co-training & one-way dis- tillation methods, with an average improvement of 0.88 and 0.47, respectively.
5.3 In-depth Study 2: Length of Training
We studied the impact of co- co-training length on the effectiveness of the CTCD method (see Figure 6). We find that longer training Ieads to improved per- formance, as demonstrated by our experiments us- ing two different training lengths: 10 epochs and 20 epochs. After pre-training the student models with these different lengths, we adapted them the CoLA downstream tasks and evaluated their performance using Matthew Correlation.
5.2 In-depth Study 1: Adjusting Loss Weights
In this Section, we investigate the impact of distilla- tion on the performance of both the student and the teacher by adjusting loss weights (at, ass BR, 3.s) for hard loss and soft loss. For example, setting Og O2 1 emphasizes learning from the teacher's knowledge (i.e., the soft loss for the stu- dent) 4x more than the ground truth label distri- bution (i.c., the hard loss for the student), during co-training. This allows us to better understand the effect of distillation on each model and optimize their performance.
Results By increasing the (co-)training length from 10 epochs to 20 epochs, CTCD (Ours) signif- icantly improves the performance of the student model from 35.02 to 46.23, with a gain of 11.21. This outperforms the conventional KD method, which only achieves gain of 0.07 from 40.94 to 41.01, despite a longer training time. The conven- tional KD method relies on pre-trained teacher model to train the student model, which allows for fast convergence but limits the learning of the student model. In contrast, the CTCD method al- lows for additional performance gains for both the teacher and student models by enabling them to learn and grow together during co-training. This can provide further benefits to the student model's performance with longer co-training.
Teacher side In Figure 5(a). we co-train the teacher model using distillation transfer knowl- edge from the student model. We fix the weighting values for the losses of the student to ay : a2 and vary the weighting values for the losses of the teacher, 3h. : 3., while training on a large text corpus. We then evaluate the average performance of the pre-trained teacher models on downstream tasks from the dev sets of GLUE benchmark.
Our results show that co-training the teacher with distillation outperforms training the teacher alone, regardless of the weighting values for the soft loss, by obtaining higher gains of 0.36, 0.77, 1.80, and 1.02 for Bn : 3g = 1 1, 2 1 : 4, and 4 4, respectively. Additionally, we find that giving more
5.4 Efficacy of Community KD
We compare the proposed Community KD with the conventional one-way KD method (Sanh et al.,

---

Downstream Task Metric
MNLI
QQP
QNLI
SST.2
CoLA
STSB
MRPC
KTE
Average
AMP
Acc.
Acc.
Acc.
Acc.
Matthww.
Pearson.
Spear.
F1
Aec.
Acc.
Acc.
Dataset Size
392.7k
363.8k
104.7k
67.3k
8.5k
5.7k
3.7k
2.5k
Teacher
BERT (109M)
84.17
90.89
57.54
89.31
65.34
One way KD (Sanh al., 2019b)
FP32
81.93
90.05
87.72
90.94
52.03
86.28
86.07
87.94
82.59
57.76
80.33
Student
Ours: Student 1
HP16
81.88
90.17
88.24
91.51
54.82
86,70
86.49
89.76
85.29
59.21
81.40
(67M)
Ours: Student
81.34
90.71
Table 2: Efficacy of Community KD The pre-trained BERT and 6-layer BERT is the teacher model and student architecture. respectively, for both ours and the conventional one-way KD method. We fine-tune the distilled students dev sets of GLUE benchmark. We observe that learning from the soft knowledge of different student model improves performance over the conventional one-way KD method
downstream tasks.
2019a). To ensure a fair comparison, we use the pre-trained BERT-base as the teacher model for both methods and the 6-layer BERT as the stu- dent, which is the same architecture used in the conventional one-way KD method. As described in Section 4, we train two student models concur- rently and they learn from the pre-trained BERT, the ground truth labels, and each other's knowledge. Note that since we fine-tune one of the two students distilled by Community KD for downstream tasks, the inference cost is the same as the conventional one-way KD method. In Table 2, we report the results of BERT and the conventional one-way KD method using checkpoints provided by Hugging Face (HuggingFace) and both students (Ours: Stu- dent 1 and Ours: Student 2) on the dev sets of the GLUE benchmark. We apply Automatic Mixed Precision (AMP) (Paszke et al., 2019) to Commu- nity KD, which typically speeds up training but may hurt performance.
teacher, it can outperform the models trained with the conventional KD on the downstream task. The proposed co-training method may increase the over- all training cost compared with one-way distilla- tion, and it may become a performance bottleneck depending on training resource constraints. How- ever, note that CTCD can improve model quality while having the same inference cost as the one- way distillation downstream tasks.
Future Work Architecture Sharing. Models can share some of their architectures by reusing the output of such architectures and updating them together during pack-propagation This may help reduce the additional computing and memory over- head incurred by model co-training, while improv- ing the model quality, especially for the student model. 2) Integration of Student Architecture Design and Data Augmentation. Future research can focus on effectively combining the CTCD frame- work with student architecture design and data aug- mentation techniques. This integration provides a promising alternative to traditional one-way knowl- edge distillation methods, lcading to significant improvements in language modeling tasks.
Results The results presented in Table 2 shows that Community KD, leads to improved perfor- mance downstream tasks such QQP, QNLI, SST-2, CoLA, STSB, MRPC, and RTE, even when applying quantization techniques. Specifically, the average performance gain of the student model dis- tilled using our Community KD method is 1.04 (1.2%) higher than that of the student model dis- tilldd by the conventional one-way KD method. This suggests that incorporating knowledge distil- lation from both student model and pre-trained teacher model is more effective than only using knowledge distillation from the pre-trained teacher model.
7 Conclusion
The size and complexity of pre-trained language models (PLMs) hinder their practicality for online downstream tasks. To address this, we in- troduced a novel framework called co-training and co-distillation (CTCD). By training models of dif- ferent sizes together and extracting inter-model knowledge in both directions, the proposed CTCD framework improves both model efficiency and per- formance. The proposed framework overcomes the trade-off between efficiency and performance in traditional one-way knowledge distillation meth- ods. Notably, our compressed models achieved an impressive gain of 1.66 on the GLUE bench- mark, outperforming large models trained using standalone methods.
6 Limitations & Future Work
Limitations
The proposed method co-train mod- els from scratch and may require longer pre- training time than the conventional KD method. However, as we described in Section 5.3, when the student model is trained long enough with its

---

References
An imperative style, high- performance deep learning library. In Advances in Neural Information Process- ing Systems 32, pages 8024-8035. Curran Associates, Inc.
Tom B Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. 2020. Language models are few-shot learners. In Neur/PS.
Yujia Qin, Yankai Lin, Jing Yi, Jiajie Zhang, Xu Han, Zhengyan Zhang, Yusheng Su, Zhiyuan Liu, Peng Li, Maosong Sun, and Jie Zhou. 2022. Knowledge inheritance for pre-trained language models. In Pro- ceedings of the 2022 Conference of the North Amer- ican Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 3921-3937, Seattle, United States. Association for Computational Linguistics.
Zihang Dai, Zhilin Yang, Yiming Yang, Jaime Car- bonell, Quoe Le, and Ruslan Salakhutdinov. 2019. Transformer-X0 XI.; Attentive language models beyond fixed-lengti context. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics. pages 2978-2988, Florence, Italy. Asso- ciation for Computational Linguistics.
Jacob Devlin, Ming-Wei Wei Chang, Kenton Lee, and
Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, llya Sutskever, et al. 2019. Language models are unsupervised multitask learners. OpenAl blog, 188):9.
Kristina Toutanova. 2019. BERT: pre-training of deep bidirectional transformers for language under- standing. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Tech- nologies, NAACL-HLL 2019, Minneapolis, MN, USA, June 2-7, 2019, Volume 1 (Long and Short Papers), pages 4171-4186.
Adriana Romero, Nicolas Ballas, Samira Ebrahimi Ka- hou, Antoine Chassang. Carlo Gatta, and Yoshua Bengio. 2015. Fitnets: Hints for thin deep nets. Proceedings of ICLR.
Wikimedia Foundation. Wikimedia downloads.
Victor Sanh, Lysandre Debut, Julien Chaumond, and Thomas Wolf. 2019a. Distilbert, a distilled version of bert: smaller, faster, cheaper and lighter. arXiv preprint 1r000:1900.01008.
Geoffrey Hinton, Oriol Vinyals. Jeff Dean, et al. 2015. Distilling the knowledge in a neural network. arXiv preprint 1eXXv:1503.025333 2(7).
Victor Sanh, Lysandre Debut, Julien Chaumond, and Thomas Wolf. 2019b. Distilbert, distilled version of bert: smaller, faster, cheaper and lighter. arXiv preprint 1910.011
HuggingFace. Huggingface.
Xiaoqi Jiao, Yichun Yin, Lifeng Shang, Xin Jiang, Xiao Chen, Linlin Li, Fang Wang, and Qun Liu. 2020.
TinyBERT: Distilling BERT for natural language un- derstanding. In Findings of the Association for Com- putational Linguistics: EMNLP 2020, pages 4163- 4174, Online. Association for Computational Lin- guistics.
Mohammad Shoeybi, Mostofa Patwary Raul Puri, Patrick LeGresley, Jared Casper, and Bryan Catan- zaro. 2019. Megatron-Im: Training multi-billion parameter language models using model parallelism. arXiv preprint arXiv: met.:-9999000000
Hayeon Lee. Rui Hou, Jongpil Kim, Davis Liang. Sung Ju Hwang, and Alexander Min. 2023. A study on know ledge distillation from weak teacher for scal- ing up pre-trained language models. arXiv preprint arXiv:2305. 18239.
Siqi Sun, Yu Cheng, Zhe Gan, and Jingjing Liu. 2019. Patient knowledge distillation for BERT model com- pression. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natu- ral Language Processing JEMNLP-IJCNLLL pages 4323-4332, Hong Kong, China. Association for Com- putational Linguistics.
Chunyuan Li, Xiang Gao, Yuan Li, Xiujun Li, Baolin Peng. Yizhe Zhang, and Jianfeng Gao. 2020. Opti- mus: Organizing sentences via pre-trained modeling of latent space. In EMNLP.
Raphael Tang, Yao Lu, Linging Liu, Lili Mou, Olga Vechtomova, and Jimmy Lin. 2019. Distilling task- specific knowledge from best into simple neural net- works. arXiv preprint arXiv: 12136.
Ilya Loshchilov and Frank Hutter. 2019. Decoupled weight decay regularization. In 7th International Conference on Learning Representations, ICLR 2019, New Orleans, LA, USA, May 6-9, 2019. OpenRe- view.net.
Henry Tsai, Jason Riesa, Melvin Johnson, Naveen Ari- vazhagan, Xin Li, and Amelia Archer. 2019. Small and practical BERT models for sequence labeling. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Lan- guage Processing (EMNLP-IJCNLP) pages 3632- 3636, Hong Kong, China. Association for Computa- tional Linguistics.
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Kopf, Edward Yang, Zachary DeVito. Martin Raison, Alykhan Te- jani. Sasank Chilamkurthy, Benoit Steiner, Lu Fang, Junjie Bai, and Soumith Chintala, 2019. Pytorch:

---

Iulia Turc, Ming- Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. Well-read students learn better: The impact of student initialization on knowledge distillation. arXiv preprint arXiv: /908,08962, 13.
Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R. Bowman. 2019. GLUE: A multi-task benchmark and analysis plat- form for natural language understanding. In Interna- tional Conference on Learning Representations.
Wenhui Wang. Hangbo Bao, Shaohan Huang, Li Dong, and Furu Wei. 2020a. Minilmv2: Multi-head self-attention relation distillation for compress- ing pretrained transformers. arXiv preprint arXiv:2012. 15828.
Wenhui Wang. Furu Wei, Li Dong, Hangbo Bao, Nan Yang, and Ming Zhou. 2020b. Minilm: Deep self- attention distillation for task-agnostic compression of pre-trained transformers. Advances in Neural In- formation Processing Systems, 33:5776-5777 5788.
Zhilin Yang, Zihang Dai, Yiming Yang, Jaime Car- bonell, Russ R Salakhutdinov, and Quoc V Le. 2019. Xlnet: Generalized autoregressive pretraining for lan- guage understanding. Advances Neural Infor- mation Processing Systems, volume 32. Curran Asso- ciates, Inc.
Li Yuan, Francis EH Tay, Guilin Li, Tao Wang, and Jiashi Feng. 2020. Revisiting knowledge distillation via label smoothing regularization. In Proceedings of the IEEF/CVF Conference Computer Vision and Pattern Recognition, pages 3903-3911.
Yukun 7hu, Ryan Kiros, Rich 7emel, Ruslan Salakhut- dinov, Raquel Urtasun, Antonio Torralba, and Sanja Fidler. 2015. Aligning books and movies: Towards story-like visual explanations by watching movies and reading books. In Proceedings of the IEEE in- ternational conference computer vision, pages 19-27.


Reviews:
[{"review_id": "RGZ68dRChU", "reviewer": "Reviewer_BAue", "summary": "", "questions": "1. what does \"training from scratch\" mean for co-training? Are the student and teacher not initialized from pre-trained BERTs? The wording is misleading and not compatible with the figures.", "limitations": "", "rating": 3, "confidence": 4, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank you for your constructive comments and for your positive feedback that **our idea is novel** and our findings are a reason to accept. We provide responses to your comments below:\n\n***\n**W1-1. The teacher’s performance improvement via the reverse KD (student —> teacher) could potentially be due to regularization effects rather than distillation.**\n- We respectfully disagree with the reviewer's comment. **KD is fundamentally a learned Label Smoothing Regularization (LSR) [1, 2, 3]**. Therefore, the regularization effect of KD should **not** be a concern; instead, it serves as **theoretical evidence** that clearly **supports** the effectiveness of our method.\n- Specifically, following theoretical analysis [1], \n  - In Label Smoothing Regularization (LSR), the goal is to minimize the cross-entropy between the smoothed label distribution $q'(k)$ and the model output $p(k)$. The smoothed label distribution $q'(k)$ is created by modifying the original label distribution: \n$$q'(k)=(1-\\alpha)q(k) + \\alpha u(k) \\tag{1}$$ which is a mixture of the original distribution $q(k)$ and a fixed distribution $u(k)$, with weight $\\alpha$, Usually, the $u(k)$ is uniform distribution $u(k)= 1/K$ where $K$ is the number of classes.\n  - The cross-entropy loss $H(q',p)$ over the smoothed labels is defined as follows:\n$$H(q',p) =-\\sum_{k=1}^K q'(k) \\log p(k) =(1-\\alpha)H(q,p) + \\alpha H(u, p) =(1-\\alpha)H(q,p) + \\alpha (D_{KL}(u, p) + H(u)) \\tag{2}$$\nwhere $D_{KL}$ is the Kullback-Leibler divergence and $H(u)$ is a constant for the fixed uniform distribution $u(k)$. Thus, the loss function of label smoothing is denoted as \n$$L_{LS} = (1 - \\alpha) H(q,p) + \\alpha D_{KL} (u, p) \\tag{3}$$\n  - On the other hand, the goal of KD is to mimic teacher behavior by minimizing the cross-entropy loss (hard loss) and KL divergence between the predictions of student and teacher as \n$$L_{KD} = (1 - \\alpha) H(q,p) + \\alpha D_{KL} (p^t_\\tau, p_\\tau) \\tag{4}$$\n  - Upon comparing Eq. (3) and Eq. (4), the sole distinction arises from the fact that $p^t_\\tau$ in $D_{KL} (p^t_\\tau, p_\\tau)$ represents a distribution derived from a teacher model, while $u(k)$ in $D_{KL}(u, p)$ signifies a pre-defined uniform distribution. In this context, we can regard KD as a special case of LSR where the smoothing distribution is acquired through learning rather than being predefined. This notion is further supported by regarding the regularization term $D_{KL}(u, p)$ as a virtual teacher model in KD, which assigns a uniform probability to all classes.\n  - Since $D_{KL} (p^t_\\tau, p_\\tau)=H(p^t_\\tau, p_\\tau) - H(p^t_\\tau)$, where the entropy $H(p^t_\\tau)$ is constant, we can reformulate Eq. (4) as \n$$L_{KD} = (1-\\alpha)H(q,p) + \\alpha H(p^t_\\tau, p_\\tau)$$\nIf we set the temperature $\\tau=1$, we have $L_{KD}=H(\\tilde{q}^t, p)$, where $\\tilde{q}^t$ is \n$$\\tilde{q}^t=(1-\\alpha)q(k)+\\alpha p^t(k) \\tag{5}$$\n  - By comparing Eq. (5) and Eq. (1), we can see more clearly that KD is a specific instance of LSR, utilizing the learned distribution $p^t(k)$ from a trained teacher rather than a uniform distribution.\n- To address the concerns from reviewer BAue's comments, we will include a discussion of the regularization effects through the distillation as the theoretical analysis to support the effectiveness of our proposed method in the revision. We appreciate the insightful comments and this discussion strengthens our paper.\n\n**W1-2. All fine-tuning is performed for 10 epochs and without early-stopping. The fine-tuning on GLUE without validation early-stopping usually has very high variances.**\n - This is a **factual misunderstanding**. As clearly mentioned in **Lines 111-120** and also depicted in **Figure 1**, our approach focused on validating the effectiveness of the proposed CTCD method in the **task-agnostic (pre-training) phase, not the task-specific (fine-tuning) phase**. In other words, in the **pre-training** phase, we conducted training for **10 epochs** (up to 20 epochs) and performed knowledge distillation. During the **fine-tuning** phase, following Hugging Face's guidelines [4], we **applied early stopping** by training all baselines, including the proposed method, for only 3 epochs (5 epochs for MRPC). Therefore, we strongly believe that our proposed method is **not** subject to the higher variance issue caused by longer training in the fine-tuning phase. \n- Nonetheless, to address the reviewer’s concern, we conducted experiments to apply our CTCD approach at the fine-tuning phase to verify the efficacy of our method. Please refer to our response for **W3**. \n- We hope that this discussion clears up the confusion of the reviewer regarding the training length of the proposed method. In the revision, we will add experimental details regarding fine-tuning and provide a clear distinction between fine-tuning and pre-training. Thanks to the discussion with the reviewer, we believe that the clarity of the paper has been improved. We appreciate the reviewer's constructive comments.\n\n***\n**W2. The performance gains over standard one-way distillation seem quite marginal based on the experiments when compared to other BERT distillation techniques like BERT-PKD, TinyBERT, MobileBERT, or BERT-of-Theseus.**\n- Directly comparing the proposed methods with BERT-PKD, TinyBERT, MobileBERT, and BERT-of-Theseus merely based on performance would be **unfair** since they have employed **extra techniques** beyond the KD algorithm for performance improvement. For example, TinyBERT uses data augmentation, MobileBERT focuses on student architecture design, BERT-of-Theseus applies progressive module replacement, and BERT-PKD conducts KD from multiple intermediate layers. **All of these extra techniques can be integrated into our CTCD algorithm orthogonally**, by replacing one-way KD in their frameworks using our CTCD algorithm instead of directly comparing them with it, thereby potentially generating additional performance improvement or synergistic effects easily.\n- Our goal is **not achieving SOTA performance, but rather verifying whether the knowledge distillation algorithm can contribute not only to model compression but also to performance enhancement**. Therefore, to clearly validate the effectiveness of our proposed method, we conducted a performance comparison between one-way KD and the proposed CTCD method in a controlled environment without employing extra techniques like architecture designs or data augmentation. Through extensive ablation studies and rigorous comparisons, we successfully demonstrated two novel findings: 1)  during co-training, distilling knowledge from the student to the teacher improves the teacher's performance and 2) the performance gains of the teacher model, in turn, benefits the student.\n- We believe that integrating our CTCD algorithm with other extra techniques is a promising and interesting research direction. We will include it as future work in the discussion section of the revision. We sincerely appreciate the reviewer’s constructive comments.\n\n***\n\n**W3. Evaluations are limited to BERT models only. Testing on other PLMs would be more convincing.**\n- As our approach has focused on the task-agnostic phase that involves training large language models from scratch on a large text corpus during the pre-training stage, training language models that are larger than BERT is challenging with limited resources from our university environment and the constraints of the rebuttal period. It would be greatly appreciated of your consideration of this situation.\n- However, we believe that the proposed algorithm is convincingly effective. As acknowledged by the **Reviewer Syen**, our extensive ablation study has supported the effectiveness of the proposed CTCD. Furthermore, by replacing solely the conventional one-way KD algorithm with the proposed CTCD algorithm at the pre-training stage **without any extra techniques**, we achieved consistent performance improvements across various downstream tasks in the GLUE benchmark on average, we attained a significant performance boost of 1.66. To see the clear efficacy of the proposed algorithm, we did not use any supplementary techniques like additional KD at the fine-tuning stage or data augmentation.\n- Furthermore, during the rebuttal period, to provide additional convincing experimental results, we conducted experiments to demonstrate the efficacy of CTCD at the **fine-tuning stage** as well as the pre-training stage. Specifically, we begin with the same pre-trained 12-layer BERT as the teacher model and the same pre-trained 6-layer BERT as the student model. Then, the proposed CTCD approach involves co-training the teacher and student models, allowing them to distill knowledge from each other for 3 epochs on a target downstream task within the GLUE benchmark. In comparison, the conventional approach of One-way Knowledge Distillation (One-way KD) follows a sequential process. It starts by independently fine-tuning the teacher model for 3 epochs without any knowledge distillation (Standalone). Subsequently, knowledge is distilled from the teacher to the student over 3 epochs. We report the final performance at the last epoch as follows:\n\n|                      |          |    MNLI    |     QQP    |    QNLI    |    SST2    |    CoLA    |    STSB    |    MRPC    |     RTE    |   Average           |\n|:------------------------------:|:---------------------:|:----------:|:----------:|:----------:|:----------:|:----------:|:----------:|:----------:|:----------:|:----------:|\n|                                |                       |    Acc.    |    Acc.    |    Acc.    |    Acc.    |  Matthew.  |  Pearson.  |    Acc.    |    Acc.    |    |\n| 12-layer   BERT      (Teacher) |       Standalone      |   83.66    |   90.77    |   91.36    |   91.46    |   58.40    |   88.21    |   84.56    |   68.23    |   82.08    |\n| 12-layer   BERT      (Teacher) | **The proposed CTCD** | **84.42** | **91.04** | **91.51** | **91.97** | **59.81** | **88.70** | **85.29** | **68.95** | **82.71** |\n|  6-layer   BERT      (Student) |       One-way KD      |   81.28    |   89.12    |   88.76    |   90.48    |   52.33    |   86.28    |   84.56    |   **58.84**    |   78.96    |\n|  6-layer   BERT      (Student) | **The proposed CTCD** | **81.67** | **90.38** | **89.55** | **91.06** | **53.44** | **86.71** | **84.80** | 58.48 |   **79.51**    |\n\n- As shown in Table, when considering the **teacher model**, the CTCD approach demonstrates clear strengths over the Standalone method. Across tasks, **CTCD consistently achieves higher accuracy and outperforms Standalone** by empirically verifying the efficacy of the CTCD approach in enhancing the teacher's performance through knowledge co-training. On the **student model** side, the comparison between One-way KD and CTCD reveals the superiority of our CTCD method. The proposed **CTCD method consistently yields better accuracy and correlation metrics** in all tasks except for RTE task, showing its efficacy in distilling knowledge bidirectionally between the teacher and student. \n- Therefore, we have successfully demonstrated the consistent strength of the proposed CTCD approach for both teacher and student models in various tasks of the GLUE benchmark, confirming the bidirectional knowledge transfer effect during co-training of the models, when compared to standalone and one-way knowledge transfer approaches, across **both the pre-training and fine-tuning stages**. We believe that including the above discussions and additional experimental results in the revision significantly strengthens our paper. Thank you for the constructive comments.\n\n**Q1. What does \"training from scratch\" mean for co-training? Are the student and teacher not initialized from pre-trained BERTs? The wording is misleading and not compatible with the figures.**\n- As we explained in **W1-2** of our response, we have applied our CTCD approach at pre-training phase (not fine-tuning phase), thus, the student and teacher are not initialized from pre-trained BERTs and they are co-trained from scratch. Therefore, the wording is correct and compatible with the figures. We will clarify this by emphasizing the pre-training phase in the revision. Thank you for your helpful feedback.\n\n[1] Yuan et al., Revisiting knowledge distillation via label smoothing regularization, CVPR 2020.\n\n[2] Tang, Jiaxi, et al., Understanding and improving knowledge distillation, arXiv 2020.\n\n[3] Kim, Taehyeon, et al., Comparing Kullback-Leibler Divergence and Mean Squared Error Loss in Knowledge Distillation, IJCAI 2021\n\n[4] https://github.com/huggingface/transformers/tree/main/examples/pytorch/text-classification\n\n***\nWe believe that including the above discussions and additional experimental results significantly strengthens our paper. Thank you for the constructive comments.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. The idea of bidirectional knowledge transfer between teacher and student during co-training is novel.\n2. The experiments show distilling knowledge from the student to teacher improves the teacher's performance, which in turn benefits the student.\n3. The student model compressed by CTCD outperforms the standalone trained original teacher model on GLUE.", "reasons_to_reject": "1. The paper studies the case where the student distills knowledge to the teacher, which improves the teacher's performance. However, the improvements could potentially be due to regularization effects rather than distillation as claimed, since all fine-tuning is performed for 10 epochs and without early-stopping. The fine-tuning on GLUE without validation early-stopping usually has very high variances, proper ablation studies are needed to verify.\n\n2. The performance gains (especially, the Community KD) over standard one-way distillation seem quite marginal based on the experiments when compared to other BERT distillation techniques like BERT-PKD, TinyBERT, MobileBERT, or BERT-of-Theseus. This is not a good signal given the training process is more complex with co-training and co-distillation compared to other distillation techniques.\n\n3. Evaluations are limited to BERT models only. Testing on other PLMs would be more convincing.", "questions_to_authors": "1. what does \"training from scratch\" mean for co-training? Are the student and teacher not initialized from pre-trained BERTs? The wording is misleading and not compatible with the figures.", "paper_topic_and_main_contributions": "The paper proposes a novel knowledge distillation framework called Co-Training and Co-Distillation (CTCD) to improve model performance while compressing large pre-trained language models (PLMs). The key idea is to co-train a large teacher model and a small student model together, enabling bidirectional knowledge transfer between them. The results on GLUE are reported. Overall, I think the idea of mutual knowledge transfer is interesting but the actual performance gains seem quite small based on the experiments.", "excitement": 3, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "FUcjWp79RY", "reviewer": "Reviewer_Syen", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely appreciate your positive feedback that our work is **well written**, **easy to understand**, **sensible**, and **performs extensive ablation studies**. We have responded your minor suggestions as follows:\n\n***\n**1. Providing implementation details for the fine-tuning phase would be beneficial.**\n- We thank you for your constructive comments. In the process of fine-tuning models, we employed the code provided by Hugging Face [1]. To ensure a fair comparison, we adhered to the guidelines outlined in the code for both the proposed CTCD and the one-way KD methods. Specifically, we employed key parameters, including a learning rate of 2e-5, and a training duration of 3 epochs for most tasks except for MRPC task and 5 epochs for the MRPC task within the GLUE benchmark. Following distilBERT [2], across all tasks, we maintained a consistent batch size of 32, accompanied by a maximum sequence length of 128. We will include the experimental details for the fine-tuning phase to improve the clarity in the revision. We sincerely thank you for your valuable and insightful suggestion once again.\n\n***\n**2. It would be interesting to explore whether this framework remains effective during a fine-tuning only stage. I would expect some experiments and analysis of this scenario.**\n- As the reviewer suggested, during the rebuttal period, we validated the proposed CTCD approach during the fine-tuning stage. Specifically, we begin with the same pre-trained 12-layer BERT as the teacher model and the same pre-trained 6-layer BERT as the student model. Then, the proposed CTCD approach involves co-training the teacher and student models, allowing them to distill knowledge from each other for 3 epochs on a target downstream task within the GLUE benchmark. In comparison, the conventional approach of One-way Knowledge Distillation (One-way KD) follows a sequential process. It starts by independently fine-tuning the teacher model for 3 epochs without any knowledge distillation (Standalone). Subsequently, knowledge is distilled from the teacher to the student over 3 epochs. We report the final performance at the last epoch as follows:\n\n|                      |          |    MNLI    |     QQP    |    QNLI    |    SST2    |    CoLA    |    STSB    |    MRPC    |     RTE    |   Average           |\n|:------------------------------:|:---------------------:|:----------:|:----------:|:----------:|:----------:|:----------:|:----------:|:----------:|:----------:|:----------:|\n|                                |                       |    Acc.    |    Acc.    |    Acc.    |    Acc.    |  Matthew.  |  Pearson.  |    Acc.    |    Acc.    |    |\n| 12-layer   BERT      (Teacher) |       Standalone      |   83.66    |   90.77    |   91.36    |   91.46    |   58.40    |   88.21    |   84.56    |   68.23    |   82.08    |\n| 12-layer   BERT      (Teacher) | **The proposed CTCD** | **84.42** | **91.04** | **91.51** | **91.97** | **59.81** | **88.70** | **85.29** | **68.95** | **82.71** |\n|  6-layer   BERT      (Student) |       One-way KD      |   81.28    |   89.12    |   88.76    |   90.48    |   52.33    |   86.28    |   84.56    |   **58.84**    |   78.96    |\n|  6-layer   BERT      (Student) | **The proposed CTCD** | **81.67** | **90.38** | **89.55** | **91.06** | **53.44** | **86.71** | **84.80** | 58.48 |   **79.51**    |\n\n- As shown in Table, when considering the teacher model, the CTCD approach demonstrates clear strengths over the Standalone method. Across tasks, CTCD consistently achieves higher accuracy and outperforms Standalone by empirically verifying the robustness of the CTCD approach in enhancing the teacher's performance through knowledge co-training. On the student model side, the comparison between One-way KD and CTCD reveals the superiority of our CTCD method. The proposed CTCD method consistently yields better accuracy and correlation metrics in all tasks except for RTE task, showing its efficacy in distilling knowledge bidirectionally between the teacher and student. \n- Therefore, we have successfully demonstrated the consistent strength of the proposed CTCD approach for both teacher and student models in various tasks of the GLUE benchmark, confirming the bidirectional knowledge transfer effect during co-training of the models, when compared to standalone and one-way knowledge transfer approaches, across both the pre-training and fine-tuning stages. We believe that including the above discussions and additional experimental results in the revision significantly strengthens our paper. Thank you for the constructive comments.\n\n***\n[1] https://github.com/huggingface/transformers/tree/main/examples/pytorch/text-classification\n\n[2] Sanh, Victor, et al. \"DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lighter.\" arXiv 201\n\n***\nThank you for your helpful comments. We believe that our paper will become much stronger after including the additional experiments you have suggested.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. The paper is is well written and easy to understand.\n2. The proposed Co-Training and Co-Distillation method is sensible, given the preliminary verification of reversed knowledge distillation.\n3. The authors perform extensive ablation studies to validate the effectiveness of CTCD.", "reasons_to_reject": "I don’t find significant flaws in this paper. There are some minor suggestions:\n\n1. Providing implementation details for the fine-tuning phase would be beneficial.\n2. It would be interesting to explore whether this framework remains effective during a fine-tuning only stage. I would expect some experiments and analysis of this scenario.", "questions_to_authors": "", "paper_topic_and_main_contributions": "This paper proposes a novel framework called Co-Training and Co-Distillation (CTCD) for knowledge distillation. This framework simultaneously pre-trains both the teacher and student models, with bidirectional knowledge distillation between the two. Experimental results on the GLUE benchmark demonstrate the superiority of this approach.", "excitement": 3, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "TKwt6JAaRz", "reviewer": "Reviewer_kfJK", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank you for your positive feedback that our work **addresses a critical limitation of KD**, is **novel** and shows two **important** empirical findings.\n\n***\n**1. I would be convinced if the authors could provide empirical analysis to demonstrate that the performance gains of the larger model are attributed to the knowledge distillation approach rather than the hyperparameter settings.**\n- Thank you for your insightful comment, and we agree that demonstrating that the performance improvement of a large model is due to the knowledge distillation process is important.\n- Fortunately, as demonstrated in **Figure 5** and **Section 5.2** of the main paper, we have already conducted experiments addressing the main concern of the reviewer. We empirically analyzed the impact of knowledge distillation from the small model to the large model based on the sizes of hyperparameter values. We observed that, **regardless of the hyperparameter sizes, knowledge distillation from the small to the larger model consistently enhances the performance of the large model**. In other words, we empirically verified that **the performance improvement of the large model is attributed to the effect of knowledge distillation, rather than the hyperparameter settings**.\n- Specifically, as shown in **Figure 5 (a)** of the main paper or the below **Table**, we investigate the effectiveness of distilling knowledge from small to large models by (co-)training the large model distilling knowledge from the small model. To this end, we fix the hyperparameters for the hard and soft (distillation) losses of the student to $\\alpha_h:\\alpha_s = 1:1$ and vary the values for the hard and soft (distillation) losses of the teacher, $\\beta_h:\\beta_s$ while training on a large text corpus. We then evaluate the average performance of the pre-trained teacher models on downstream tasks from the GLUE benchmark.\n    \n|         | Stand-alone (w/o KD) |  $\\beta_h:\\beta_s=1:1$  |  $\\beta_h:\\beta_s=1:2$  |  $\\beta_h:\\beta_s=1:4$  |  $\\beta_h:\\beta_s=4:4$  |\n|:-------:|:--------------------:|:-----:|:-----:|:-----:|:-----:|\n| Average |         78.06        | 78.42 | 78.83 | 79.86 | 79.09 |\n|   Gain  |           -          | +0.36 | +0.77 | +1.80 | +1.02 |\n    \n- Our results demonstrate that the distilled large model achieves significantly greater performance improvements of 0.36, 0.77, 1.80, and 1.02, respectively, compared to the independently trained large model (Stand-alone) with a performance of 78.06, for **all cases** of $\\beta_h:\\beta_s = 1:1, 1:2, 1:4$, and $4:4$, consistently. (For a more detailed explanation, please refer to Section 5.2.)\n- Interestingly, we additionally observe experimental results that partially support the reviewer’s argument: larger hyperparameter values ($\\beta_s$) for soft losses can lead to longer training effects. Specifically, giving more weight to the soft loss of the larger model during training ($\\alpha_h:\\alpha_s:\\beta_h:\\mathbf{\\beta_s} = 1:1:1:\\mathbf{1} \\rightarrow 1:1:1:\\mathbf{2} \\rightarrow 1:1:1:\\mathbf{4}$) brings about performance improvement. The average scores increased from $78.42$ to $78.83$, and further to $79.86$. However, we observe that a larger hyperparameter value for the hard loss ($\\beta_h$) does not necessarily lead to a longer training effect. Emphasizing only the soft loss of the teacher (1:1:1:4) actually results in better performance compared to emphasizing both the hard and soft losses of the teacher (1:1:4:4), with average scores of 79.86 vs. 79.09.\n- We believe that this finding is a meaningful analysis that contributes to a better comprehension of the proposed method and further strengthens the effectiveness of our method. In other words, knowledge distillation from the small model to the large model brings performance improvements ranging from 0.36 to 1.80 in all cases when compared to independently training the larger model, due to the effect of knowledge distillation. Furthermore, increasing the hyperparameter values for soft losses ($\\beta_s$) allows for reducing the training time of the larger model or achieving a greater distillation effect within a limited training time. However, increasing the hyperparameter values for hard losses ($\\beta_h$) does not yield such effects.\n- Additionally, we provide a theoretical analysis of the enhancement in the teacher's performance achieved by distilling knowledge from the student in **W1-1** of reviewer BAue. For more details, please see our response to W1-1 of reviewer BAue.\n- We will add this valuable discussion in the 5.2 section of this paper, highlighting the effect of increasing hyperparameter values for soft losses that can potentially shorten training time for the larger model, leading to a larger distillation effect. We hope that this result addresses your concern and the discussion strengthens our paper. We thank you so much for your helpful and insightful suggestion.\n\n***\n**2. Using 6-layer BERT as the teacher and 4-layer BERT as the student seems like an unusual configuration. A more reasonable setting would be to use 12-layer BERT as the teacher and 6-layer or 4-layer BERT as the student model.**\n\n- Thank you for the valuable feedback from the reviewer. To address the concerns raised during the rebuttal process, we conducted experiments in a typical setting using a 12-layer BERT as the teacher model and a 6-layer BERT as the student model to demonstrate the effectiveness of the proposed CTCD approach.\n- Due to limited resources in our university environment and the constraints of the rebuttal period, we focused on validating the proposed CTCD approach during the fine-tuning stage, instead of resource-intensive pre-training, which would have required training a large model on a large text corpus from scratch. It would be greatly appreciated of your consideration of this situation.\n- Specifically, we begin with the same pre-trained 12-layer BERT as the teacher model and the same pre-trained 6-layer BERT as the student model. Then, the proposed CTCD approach involves co-training the teacher and student models, allowing them to distill knowledge from each other for 3 epochs on a target downstream task within the GLUE benchmark. In comparison, the conventional approach of One-way Knowledge Distillation (One-way KD) follows a sequential process. It starts by independently fine-tuning the teacher model for 3 epochs without any knowledge distillation (Standalone). Subsequently, knowledge is distilled from the teacher to the student over 3 epochs. We report the final performance at the last epoch as follows:\n\n|                      |          |    MNLI    |     QQP    |    QNLI    |    SST2    |    CoLA    |    STSB    |    MRPC    |     RTE    |   Average           |\n|:------------------------------:|:---------------------:|:----------:|:----------:|:----------:|:----------:|:----------:|:----------:|:----------:|:----------:|:----------:|\n|                                |                       |    Acc.    |    Acc.    |    Acc.    |    Acc.    |  Matthew.  |  Pearson.  |    Acc.    |    Acc.    |    |\n| 12-layer   BERT      (Teacher) |       Standalone      |   83.66    |   90.77    |   91.36    |   91.46    |   58.40    |   88.21    |   84.56    |   68.23    |   82.08    |\n| 12-layer   BERT      (Teacher) | **The proposed CTCD** | **84.42** | **91.04** | **91.51** | **91.97** | **59.81** | **88.70** | **85.29** | **68.95** | **82.71** |\n|  6-layer   BERT      (Student) |       One-way KD      |   81.28    |   89.12    |   88.76    |   90.48    |   52.33    |   86.28    |   84.56    |   **58.84**    |   78.96    |\n|  6-layer   BERT      (Student) | **The proposed CTCD** | **81.67** | **90.38** | **89.55** | **91.06** | **53.44** | **86.71** | **84.80** | 58.48 |   **79.51**    |\n\n- As shown in Table, when considering the teacher model, the CTCD approach demonstrates clear strengths over the Standalone method. Across tasks, CTCD consistently achieves higher accuracy and outperforms Standalone by empirically verifying the robustness of the CTCD approach in enhancing the teacher's performance through knowledge co-training. On the student model side, the comparison between One-way KD and CTCD reveals the superiority of our CTCD method. The proposed CTCD method consistently yields better accuracy and correlation metrics in all tasks except for RTE task, showing its efficacy in distilling knowledge bidirectionally between the teacher and student. \n- Therefore, we have successfully demonstrated the consistent strength of the proposed CTCD approach for both **12-layer BERT teacher and 6-layer BERT student models** in various tasks of the GLUE benchmark, confirming the bidirectional knowledge transfer effect during co-training of the models, when compared to standalone and one-way knowledge transfer approaches, across **both the pre-training and fine-tuning stages**. We believe that including the above discussions and additional experimental results in the revision significantly strengthens our paper. Thank you for the constructive comments.\n\n***\nOnce again, we extend our sincere appreciation for your time and dedication in reviewing our paper and we strongly believe that our paper be strengthened thanks to the reviewer's valuable comments.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. This paper addresses a critical limitation of knowledge distillation (KD) methods - the tradeoff between model compression and performance degradation.\n2. This paper proposes a novel co-training and bi-directional distillation approach. Most prior work uses one-way distillation which loses knowledge. In contrast, CTCD allows mutual transfer between teacher and student models.\n3. Experimental results indicate two important empirical findings related to co-distillation: 1) Distilling knowledge from smaller to larger model boosts the larger model's performance. 2) The improved larger model further enhances the smaller model's performance. Additionally, the distilled small model significantly outperforms the original large model by 1.66 on GLUE benchmark.", "reasons_to_reject": "1. My major concern is about the source of the performance improvement, especially the finding that distilling knowledge from smaller to larger models boosts the larger model's performance. I am wondering if the improvement could be attributed to the values of the hyperparameters α_h, α_s, β_h and β_s. Although the training epoch is 10 for all different settings, larger values for these hyperparameters result in longer training. I would be convinced if the authors could provide more empirical analysis, e.g. ablation studies or theoretical analysis, to demonstrate that the performance gains are due to the distillation approach rather than the hyperparameter settings.\n\n2. Another key concern is the selection of teacher and student models. The authors mentioned using 6-layer BERT as the teacher and 4-layer BERT as the student. This seems like an unusual configuration. A more reasonable setting would be to use 12-layer BERT as the teacher and 6-layer or 4-layer BERT as the student model.", "questions_to_authors": "", "paper_topic_and_main_contributions": "Knowledge Distillation (KD) compresses large pre-trained language models by transferring their knowledge to smaller models, enabling use in resource-constrained settings, but often sacrifices performance. To address this, the authors propose Co-Training and Co-Distillation (CTCD), a framework that co-trains two models of different sizes, mutually distilling knowledge between them. CTCD improves both model performance and inference speed, based on findings that: 1) Distilling knowledge from the smaller to the larger model boosts the larger model's performance, and 2) The improved larger model further enhances the smaller model's performance. CTCD shows promise to combine with techniques like architecture design and data augmentation, replacing one-way KD, for further gains. Extensive ablation studies demonstrate CTCD's effectiveness, where the small model distilled by CTCD significantly outperforms the original larger model by 1.66 on GLUE benchmark.", "excitement": 3, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper proposes a novel distillation framework where student and teacher models are co-trained to enable superior transfer between them.  The framework is novel and the paper is well written. The rebuttal has acknowledged some of the limitations around the experimental setting (instantiations of the student and teacher models). The performance gains are small but significant. I think the community would benefit from this novel framework.
Decision: Accept-Findings