A Good Learner can Teach Better: Teacher-Student Collaborative Knowledge Distillation
Authors: Ayan Sengupta
Shantanu Dixit
Md Shad Akhtar
Tanmoy Chakraborty
Venue: ICLR 2024 poster (2024)
Keywords: Knowledge Distillation
Meta-Knowledge Distillation
Policy-driven Knowledge Distillation
Large Language Models
Primary area: transfer learning, meta learning, and lifelong learning
TL;DR: The paper introduces collaborative joint loss and curriculum learning for meta-teacher knowledge distillation

Abstract: Knowledge distillation (KD) is a technique used to transfer knowledge from a larger ''teacher'' model into a smaller ''student'' model. Recent advancements in meta-learning-based knowledge distillation (MetaKD) emphasize that the fine-tuning of teacher models should be aware of the student's need to achieve better knowledge distillation. However, existing MetaKD methods often lack incentives for the teacher model to improve itself. In this study, we introduce MPDistil, a meta-policy distillation technique, that utilizes novel optimization strategies to foster both *collaboration* and *competition* during the fine-tuning of the teacher model in the meta-learning step. Additionally, we propose a curriculum learning framework for the student model in a competitive setup, in which the student model aims to outperform the teacher model by self-training on various tasks. Exhaustive experiments on SuperGLUE and GLUE benchmarks demonstrate the efficacy of MPDistil compared to $20$ conventional KD and advanced MetaKD baselines, showing significant performance enhancements in the student model -- e.g., a distilled 6-layer BERT model outperforms a 12-layer BERT model on five out of six SuperGLUE tasks. Furthermore, MPDistil, while applied to a large language teacher model (DeBERTa-v2-xxlarge), significantly narrows the performance gap of its smaller student counterpart (DeBERTa-12) by just $4.6$% on SuperGLUE. We further demonstrate how higher rewards and customized training curricula strengthen the student model and enhance generalizability.

Published as a conference paper at ICLR 2024
A Good Learner can Teach Better: TEACHER-STUDENT COLLABORATIVE KNOWLEDGE DISTILLATION
Ayan Sengupta', Shantanu Dixit Md Shad Akhtar?, Tanmoy Chakraborty'
'Indian Institute of Technology Delhi, India
Indraapasttha Institute of Information Technology Delhi, India
ayan. seee wiiiiiiaa.. ac. in, (shantanu20118, shad. t lanchak@iitd.. in
ABSTRACT
Knowledge distillation (KD) is technique used to transfer knowledge from larger "teacher" model into a smaller "student" model. Recent advancements meta-learning-bssed knowledge distillation (MetaKD) emphasize that the fine- tuning of teacher models should be aware of the student's need to achieve better knowledge distillation. However, existing MetaKD methods often lack incentives for the teacher model to improve itself. In this study, we introduce MPDis 1, meta-policy distillation technique, that utilizes novel optimization strategies foster both collaboration and comperition during the fine-tuning of the teacher model in the meta-learning step. Additionally, we propose curriculum learning framework for the student model in a competitive setup, in which the student model aims to outperform the teacher model by self-training on various tasks. Exhaustive experiments on SuperGLUE and GLUE benchmarks demonstrate the efficacy of MPDistil compared to 20 conventional KD and advanced MetaKD baselines, showing significant performance enhancements in the student model e.g., a distilled 6-layer BERT model outperforms a 12-layer BERT model on five out of six SuperGLUE tasks. Furthermore, MPDist i1, while applied a large language teacher model (DeBERTa--2-xxlagge) significantly narrows the performance gap of its smaller student counterpart (DeBERTa- 12) by just 4.6% on SuperGLUE, We further demonstrate how higher rewards and customized training curricula strengthen the student model and enhance generalizability.
INTRODUCTION
Large language models (LLMs) like GPT-3 (Brown et al.) 2020), PaLM (Chowdhery et al. 2022)
and LLaMA (Touvron al. 2023) have demonstrated remarkable performance in wide range of natural language tasks, showcasing their abilities tasks requiring minimal, zero, or context-based prior knowledge (Kossen et al. 2023). These models underwent extensive pre- mre-training datasets to achieve such impressive capabilities, incurring substantial computational costs, as high- lighted by Zhhu et al. (2023). The scaling laws associated with language models (Kaplan al. 2020) indicate that achieving superior predictive performance comes at the cost of a large number of model parameters, extensive training data, and substantial computational power. However, these costs typi- cally do not account for inference expenses, which must be handled by resource-constrained devices when deploying these models at scale. One approach address this challenge is "model compres- sion" (Deng et al. 2020). technique designed to reduce computational requirements and storage while preserving the original models' performance.
massive
Knowledge distillation (KD) (Hinton et al. (015)) is a widely adopted model compression method that entails transferring knowledge from a larger and more complex model, referred to as the "teacher model", to smaller and simpler model known
the "student model". In traditional white-box knowledge distillation, the primary focus is on minimizing the "teacher-student margin" by narrow- ing the gap between the teacher's distribution, fo (y)xr) (z). and the student's distribution, so (yylw). More advanced knowledge distillation techniques (Sun et al. 2019 Jiao et al. (2119) go further by aligning the intermediate feature maps between the teacher and student models. This alignment helps the stu- dent replicate the behaviors demonstrated by the teacher model. However, most of these techniques operate in unidirectional manner, in which the teacher model remains fixed after pre-training or

---

Published as a conference paper at ICLR 2024
fine-tuning, while the student model is trained to minimise the teacher-student margin. Zhou et al. (2021) pointed out several limitations of this approach. Existing approaches do not optimize the teacher explicitly for the distillation task or consider the student model'ss capacity during training. This differs from the real-life teacher-student dynamic, in which a teacher continuously enhances her knowledge and teaching skills through repeated student interactions. To address this gap, ZZou al.] (222)) introduced MetaDistil, meta-learning- based knowledge distillation framework that aims to overcome this limitation by considering the student's predictive performance while fine-tuning the teacher model. In MetaDistil, the teacher model undergoes training on a separate quiz dataset to minimize the knowledge distillation loss between the teacher and student models along with the teacher's task-specific loss. Although MetaDistil addresses several shortcomings of conventional KD methods, it is still far from imitating real-world teacher-student interactions. In a real-world educational context, teachers may not solely focus narrowing the teacher-student knowledge gap to improve teaching; instead, they aim to maximize the joint performance of students and teachers. Furthermore, in typical classroom setting, students study various subjects and follow curricu- lum to optimize their learning across all subjects. For example, a student might aim to improve her understanding of Physics by studying selected concepts from Mathematics. However, existing knowledge distillation methods tend to distil knowledge for different tasks in isolation, which means they fail capture the commonalities among tasks. Consequently, the distilled models struggle to generalize effectively to new tasks with limited training data. Another significant challenge with the sole optimization of the teacher-student margin is that it positions the teacher as the "sole role model" for the student without encouraging the student to surpass the teacher's ' performance.
To address these limitations, we introduce MPDistil 11, a meta-policy knowledge distillation frame- work that employs collaborative learning approach within the context of meta- keta-knowwedge distil- lation] Through empirical analyses, we underscore that optimizing a shared utility function can enhance the teacher's predictive capabilities and ability to impart knowledge to the student model. Furthermore, we establish a meta-reinforcement learning-based "curriculum learning" paradigm in which the student model undergoes fine-tuning to enhance the student model's generalization skills, enabling it to surpass the teacher model. This competitive approach empowers the student outperform the teacher. We assess the effectiveness of MPD st i.l using two natural language understanding benchmarks, SuperGLUU (Wang et al., 2019) and GLUE (Wang al., 2018). encompassing 15 NLU tasks. Following the experimental setup adopted by contemporary KD-related studies, we highlight the effectiveness of MPDi stil with BERT-base (Devlin et al. 2018) as the teacher model and BERT 6L (with six layers) as the stu- dent model. On the SuperGL UEE benchmark, the meta update enhances the BERT teacher's 's perfor- mance with a maximum margin of +3%. resulting notable improvement in student performance by +5.9%. Surprisingly. the distilled student surpasses the teacher model on five of six SuperGI.UF tasks within this competitive framework, with a maximum advantage of + 7%. contrast, the most competitive baseline, MetaDistil, exhibits average deficit of - -.11% indicating that the distilled student falls short of the teacher model's performance. We also extend our evaluation to a large lan- guage model, DeBERTa-V2-xllal DeBERTa-12 12, as the student model. Ours is the first large-scale study highlighting the effective- ness of distilling knowledge from large language models (> 1B parameters) to smaller models. uPDistil 1, when applied to DeBERTa, improves the student model's performance with a maxi- mum margin of +2.2%, reducing the student-teacher margin to -4.6% (a negative student-teacher margin indicates that student underperforms the teacher), which is significantly higher than the other distillation methods (-7.%%).
curriculum sequence of tasks. The aim is
(He et al. 2020), the teacher model and its smaller variant,
2 RELATED WORK
Knowledge distillation. One of the earliest investigations on knowledge distillation by Hinton et al. (2015) introduced technique in which the student model is trained to minimize the task task-specific loss while also enhancing the similarity between the outputs of the teacher and student models. This enhancement is achieved by minimizing the difference between softened class probabilities generated by both the teacher and student model using temperature scaling. Sanh et al. (2019) distilled a more compact version of BERT-base by halving the number of layers and removing the pooler output and token type embeddings. In contrast to learning solely from the teacher's
Source code of MPDİ st: be found at httppss//ggccccb.ccoo///o/////tt
6 6/MPDi st
2

---

Published as a conference paper at ICLR 2024
last layer, [Sun al. (2019) introduced "patient learner" approach. The student learns from the last k layers by incorporating a Mean Squared Error (MSE) loss between the teacher and student layer representations. leveraging the internal representations of the teacher model. TinyBERT (Jiao (et al. 2019) also utilises the internal representations of BERT to distil knowledge. It achieves this by distilling the transformer, embedding, and classification layers. Yang al.. (2020) and WW & Chiu (2020) introduced multiple teacher-based knowledge distillation each teacher is assigned predetermined weights, and their weighted average probability distributions are used to train the student model. In a similar attempt. Yuan et al. (202T) proposed reinforcement learning- baased technique in which different weights are assigned to various teacher models during distillation using policy function.
Meta knowledge distillation. Meta-learning-basee knowledge distillation techniques utilize a meta-learning loop to enhance the teacher model. Established MetaKD approaches (Pan et al. [2020]
Liu et al. 2022b) employ meta-teachers to adapt them to new domains and tasks. [Liu et al. (2022b) argued that using fixed temperature can result in suboptimal knowledge transfer from teacher to student. They aimed to meta-learn the student and teacher's temperature along with the distillation function. In contrast, Zhou al. (2021) introduced a generalized framework, MetaDistil, focus- ing on teaching teachers how to facilitate knowledge distillation. Note that these techniques entail fine-tuning meta-teacher, which is computationally equivalent the teacher model. Fine-tuning a large pre-trained teacher model can be prohibitively expensive. Furthermore, the meta-learning objective optimized by the meta-teacher primarily aims narrow the gap between the teacher and student models without encouraging them to surpass each other.
Curriculum based knowledge distillation. Curriculum learning (Bengio al. 2009 involves increasing the level of training samples progressively during training. Xiang et al. 2020) designed curricula based on the confidence score of the mixture of expert models, which was later used to train a unified student model. [Li et al.) (2023) proposed learning a curriculum from easy to hard levels through a learnable temperature optimized through a reversed gradient. involved increasing the distillation loss through a cosine schedule, leading to an increased difficulty for student learning.
Our work introduces collaborative and competitive utility optimization within the meta-knowledge- distillation paradigm and establishes competitive reward system for the student using task based curricula. This competitive framework motivates both teacher and student models to outperform each other. The joint utility can also be optimized using a simpler inating the need to fine-tune the teacher backbone. This enhances the generality of our proposed method, making it suitable for distilling large pre-trained models and making method more computationally efficient.
meta-teacher, elim-
PROPOSED METHODOLOGY
Our proposed meta-policy knowledge distillation method, MPDistil, aims to distil knowledge from arbitrary teacher model, parameterized by 0r to an arbitrary student model, parameter- ized by 0s. Although the teacher and student models can belong to any family (c.g., encoder, encoder-decoder or decoder), we typically assume that they belong the same architectural family. Additionally, we assume that both the teacher and student models can be extended to a multi-task framework, wherein the models can be optimized on any task T E T. MPDistil consists of four steps (i) fine-tuning the teacher model, (ii) distilling the teacher model to the student model, (iii) meta-teacher learning, and (iv) student curriculum learning. We illustrate these steps in Figure
Fine-tuning teacher model. For any given task T E T, we use the task-specific loss function Ly optimize and fine-tune the teacher model. For classification tasks, cross-entropy is used as the task-specific loss, whereas for regression tasks, MSE is used to calculate the loss. Precisely, for a given training data batch of size N, Dtrain = {(11111)) 11), (xN,yN), )x)), we calculate Ly teacher = N EM LT(1, )) with ((rT)) = T(x;; 0r), the output generated by the teacher model T with parameters 8r. The teacher model parameters are then updated with gradients computed on Cy with respect 0r. Although highlight the teacher fine-tuning with vanilla training procedure, for large language models, it can be replaced with parameter-efii zero-shot fine-tuning.
fie-tunning or even few-shot
3

---

Published as a conference paper at ICLR 2024
Step Teacher fine-tuning
Step Student distillation
Nomendature
Studont modl
Task
Task J Train
Ys
Lij
Extudent KD
Carrinaiun made
Train
Yr
Forward pas
Backward pass
Cteacher
forward without gracher updele
Step 4: Student curriculum learning
Step 3: Meta-teacher learning
Teucher
hidden representationn
hidden representation
Student hidden representation
Student
Taak
Tisk Cuse
Quiz
saadest
hidden ropresentaaioo
reward
REINFORCE
Figure I: A schematic diagram of our proposed MPDistil framework. It consists of four steps teacher fine-tuning step, where the original teacher model is fine-tuned on task T, (ii) student distillation step, where the student model is fine-tuned with task-specific and knowledge distillation loss, (iii) meta-teacher learning step, where a feed-forward model, the meta-teacher, is trained to jointly optimize the outputs for the teacher and student hidden representations, and (iv) in student curriculum learning step, we train a student curriculum model to generate suitable curriculum, a sequence tasks T' E for which the distilled student fine-tuned and calculated rewards.
Distilling teacher model to student model. In step 2 of our framework, we distil the student model S with the knowledge acquired by the teacher model with fine-tuning. We use the task- specific loss fT along with a knowledge distillation loss LXD computed on the training dataset. MPDist iitistill 1 is a general framework and can handle any arbitrary LKD function, as long as it is differentiable with respect to the student model parameters fg. Following Zhou et al. (202T), we use convex combination of the MSE loss between the teacher and student logits and the mean squared differences between the teacher and student hidden representations, Formally, we calculate,
N aLr(u, ((.ss) + (1 - a)
C student
(..T)
((.)))
+ B.b. h(..s) ||
N
Here, T denotes the Softmaz activation function. Although some studies (Hinton et al., 2015 Gu et al. 2023) use Kullback-Leibler Divergence (KLD) and reverse KLD between the teacher and student logits, following Kim et al. (2021). we use L2 norm to calculate the difference between teacher and student logits, to stabilize the optimization. Hence, a student optimized with Ly student not only trained the task T standalone but also acquires the knowledge from the teacher through soft label association and shared latent, hidden representations.
Meta-teacher learning. In step 3 of MPDistil, we jointly update the teacher model parameters optimize the teacher and student performances. To achieve this, we devise meta-teacher T', parametrized with 0r. The meta-teacher follows a simple feed-forward network architecture with the ability to digest the hidden state representations from both the teacher and student models and generate the final output. In our white-box distillation framework, we assume that the output logits and hidden representations are accessible from both the teacher and student models. The meta- teacher, therefore, can act discriminator, separating the representations extracted by the original teacher and the student models. We train the meta- tetahee parameters 8, as the meta-learner, whereas the student model parameters 0g are learned the inner-learner. It is interesting notice that as opposed to the meta-teatt teacher designed by Zhou et al. (202T), our meta-teeccerr has only an insignificant number (only 0.001 % of the teacher model) of learnable parameters and, therefore,
4

---

Published as a conference paper at ICLR 2024
much easier to train in the meta-learning setup. We avoid large pre-trained teacher model as cloning it to a meta-teacher and fine-tuning could be prohibitively expensive. Our meta-teacher is, therefore, computationally more efficient, even for distilling from computationally large pre- ee-trrined models.
We adopt quiz dataset QT for each task T, separate from the original training dataset for meta- teacher learning. We formulate two different training objectives for the meta-teacher. In the first approach, compute collaborative loss, in which the meta-teacher optimizes the joint task- specific losses calculated with the teacher's and student's hidden representations. Given a training data (Ig ys), we obtain the hidden states from the teacher and student models as h(L,T) and ((1,)) respectively, and obtain the final output from the meta-teacher model as, ((G.)) = T (harr); and (1.s)) T' (M(s.)); (pr)) respectively. If the task T is classification task, we use only the class probabilities of the true class V(a.T) (((,TT)) and V(i.s) (((,,))) for arg max (((,))) the true class label. The collaborative loss is formally defined as,
log M)) + log ((s.s)
i' a classification task
LT meta col
2N
-/100- wー b1.0) i regression task
The meta-teacher acts discriminator and does not understand whether the teacher the stu- dent obtains the hidden representation. Moreover, the original student and teacher parameters are frozen during meta-teacher training, and the meta-teacher trained. Therefore, the meta-teacher only learns the optimal function to jointly optimize the teacher's and student's final predictive per- formances. We compute competitive loss in another formulation between the teacher and student representations. The competitive loss is motivated by the Wasserstein loss used in WGAN (Arjovsky) al. 2017) and calculates the Earth Mover's (EM) distance between the teacher and student logits. Additionally, we add the meta-teacher's 's task-specife loss so that the meta-tes meta-teacher can minimize its loss. along with maximizing the margin from the student model. Using the Miattoovvic-RRueensteee duality (Villani et al. 2009), we compute the competitive loss as,
N
Σ
2 log D(,,7) log D(s.8)
if Tis classification task
Cy meta com
N
(3)
S..ml ll.
= SC..5 l|
if Tis regression task
Proposition For any classification task T. having B meta c student with [[[ss] E(yr]
< E[ meta ensures stronger
Student curriculum learning. The final step of MPDistiL is the student curriculum learning, in which the student model aims to outperform the meta-teacher by training itself on a suitable set of tasks, possibly different from the task hand T. Curriculum learning (Bengio et al.) (009) involves gradually acquiring skills by initially focusing on simpler tasks before progressing to more complex ones. We use policy learner, a curriculum model, learn the curriculum for the student. The curriculum model is feed-forward network that takes the current state of the student model and samples a task (action) T T in each forward pass. The student model is trained on batch sampled from Q and is used to calculate reward on QT. For an updated student model, we calculate the output Ú(S.s) = ((;;;0s) for each (æi,, W:) ~ QT. Similarly, we obtain the output "(rr")) from the meta-teacher. Analogous to the previous meta-teacher learning step, we obtain only the predicted probability of the true output class for classification tasks. We evaluate two different reward functions: (i) binary reward and (ii) real reward.
if Tis is a classification task
R hinary
||. | (c..5
if Tis regression task
(4)
9cc.5)
2 Alll the proof' presented the paper are supplied Appendix

---

Published as a conference paper at ICLR 2024
Here, II is the indicator function. Similarly,
Div.s) Berr' ||u. -uerr 112 Becss)
if Tis classification task
R real
(5)
if Tis a regression task
Proposition 2. For any classification task T. E binary] E|
The expected reward for an entire episode is calculated as G, = Ek= - Ra. with a suit- able discount factor 7 and the length of the episode F. In our case, the length of episode is the number of quiz batches for task T. We use the Monte Carlo policy gradient algorithm, RE- INFORCE (Williams] 1992; Sutton al.,) 1999), to update the curriculum model parameters 04 to maximize the expected reward. The curriculum model samples tasks that maximize the reward for the student model, i.e., increases the difference between the student and teacher models. In other words, having trained on these newly sampled tasks, the student model can achieve better results than the teacher when evaluated through meta-teacher, leading to higher rewards.
4 EXPERIMENTAL DETAILS AND RESULTS
Following the contemporary knowledge distillation studies (Liu et al. 2022a Zhou et al. 2021 evaluate MPDist iisiiiiii on 15 different natural language understanding tasks from SuperGI.URi (Wang) ctt al., 2019) and GLUE (Wang et al. 2018)) benchmarks. We choose the classification tasks CB, COPA, RTE, WiC, WSC and BoolQ from the SuperGLUE benchmark. From the GLUE bench- mark, we use MNLI, MNLI Mismatch, QNLI, QQP, WNLI, MRPC, RTE, SST-2 and STS-B tasks, out of which only STS-B is regression task and the remaining are classification tasks. We elab- orate on these tasks in Appendix B.I We utilize the pre-trained BERT-base (Devlin et al., 2018) model as the teacher (111M parameters) and the pre-trained BERT 6-layer model the student Additionally. we evaluate MPDistiL on a pre-trained large language model,
) we
(66M parameters 3 DeBERTa-v2-xxlarg (He et al.) (2020) (1.4B parameters) and its 12-layered variant (547M param- eters) as the student model. As described in Section for teacher, meta-teacher and student models, we use multiple output heads depending on the number of tasks. For SuperGI.UF and GLUE, we use six and nine output heads (one for each task), respectively. However, all the models are trained on each task separately. The BERT-base model utilizes a pooling layer for learning suitable pro- jection from the output representation corresponding to the first token ([CLS] token) and uses it for downstream tasks. We use the student model pooling layer weight matrix and learn projection vector on 768 to represent the state of the student model in curriculum learning. Meta-teacher learning and student curriculum learning require a separate labelled dataset, for which we split the original training dataset provided in SuperGI.UP and GLUE tasks into updated training and quiz sets by 9 : 1. However, we use the original datasets provided in the benchmarks for dev and test. We furnish other experimental details in Appendix B.2 We evaluate MPDistil with 20 contemporary KD and MetaKD methods, some of which are claborated in Appendix B.3
We report the performances of different models - the fine-tuned teacher model, the dis- tilled student, the meta-teacher and the updated student with MPDi st 1 on the SuperGL.UF tasks in Table [ Additionally, we report the results obtained from the student model when fine-tuned on the tasks without distillation. As ablation to MPDistil alternate student model that is only trained with LKD in step (c.f. Figure I. without curriculum learning. We denote this model as '(-) Curriculum learning". On the SuperGLUE dev split, the meta- teacher obtains 0.7% better performance than the original teacher model. Although both the loss for- mulations are adequate for the meta-teacher, collaborative loss helps the meta-teacher obtain better performance, as opposed to the competitive loss. Collaborative loss is equally essential for better stu- dent distillation. On SuperGLUE dev, the student distilled with collaborative loss and binary reward achieves 5.9% and 1.5% better than the base student and teacher models, respectively. Although both reward functions are equally effective, the distilled student achieves 0.2% improvement with binary reward than with the real reward function. Without student curriculum learning, the ablation model achieves 2.4% better than the base student model. However, the improvement over the teacher model remains muted at - 2%11% indicating the positive impact of curriculum learning the student.
Results.
evaluate the performance of an
"The pre-trained models are obtained from http:::////........ //tp:///uggggg// wtt.:/cccccccoocaaacoo s/

---

Published as a conference paper at ICLR 2024
BeolQ
WIC
Mechods
ReolQ
COPA
Toacher (BEX))a
15.5
$3.9
65.0
57:1
57.1
65.4
Toaher (bBBid
56.8
92.1
$4.0
90.6
53.3
63.4
Stateor (BERT (1)
71.6
75.0
53.0
64.6
56.0
63.5
Stadeer (D 121)
12.1
82.1
64.0
79.4
57.
63.5
Dutlial Sualert
25.0
50.6
56.0
56.4
Ditiled Suuder
$2.4
750
02.0
76.1
91
SLL
Maa fa
Mil ma
Cul
755
83.9
59.5
(4)
36.6
53.0
19.5
63.5
Stafeer wis 1T
Sia.i with NP
(1) Oul loss Binayy ceward
12.8
$3.9
67.0
57:1
58.0
65.4
(1) Cal loss Bingy aewud
82.7
76.8
67.0
79.8
03.0
66.3
(+) Oall line + Real rowoed
9%.6
$2.1
70.0
56.4
58.6
64.4
(+) Cal lane Real soseed
$2.7
76.7
67.0
78.0
90.0
65.4
(8) Can los Banay reward
75.0
$0.4
62.1
67.5
59.6
05.4
(1) Com loss Basay reoad
82.4
76.8
08.0
80.
59.3
65.4
(+) Oum
moveml
78.6
56
(+) Clem Bead poward
76.8
68.0
77.8
64.4
1-) Caniculun leamnng- Comp
72.5
72.3
58.0
Carrzutim
loss
64.0
80.
64.4
(a) BERT Teacher
(b) DeBERTa Teacher
Table 1: Results of our method (MPDistil) on the SuperGLUE dev set with the BERT and De- BERTa teacher models. We report the accuracy scores for each of these classification tasks. We observe the teacher and student base models trained on the original train set and compare them against the distilled student and the meta-teacher model. We highlight the cases in blue in which the meta- outperforms the original teacher model. Scenarios highlighted with brown are the cases in which the distilled student model performs better than the teacher.
To assess the effectiveness of MPDi il in distilling knowledge from the teacher to the student models, we compare it with other competitive knowledge distillation
Methods
BoolQ
CB
COPA
RIE
WIC
WSC
KD Fintoon et
2015
-13.3
-19.1
-4.3
-3.7
-9.1
-14.4
PD Ture sta
2013
-9.5
-0.3
-13.5
-6.9
-11.2
-12.5
DeniBERTTsmm
2019
-7.7
-12.0
.9.3
Thesans Nuterm buza
-1.6
-36
-43
-48
-11.5
techniques.
kd19
-1.4
-1.2
4.3
-3.7
1.7
-29
As the performance of the distilled student depends on the teacher's perfor- mance, we derive an evaluation metric, Aarggin = (Performance of student Performance of teacher). to compare the distillation methods. The metric indicates the student's performance in terms of the teacher (or the im- provement over the teacher), which is agnostic to the original teacher's performance. It worth noting that existing distillation techniques can not empower the student model to perform better than the teacher model, often leading to a negative Margin. We report AMargin for different distillation techniques on the SuperGLUE dev in Table 2 (with BERT models) and Table 3 (with DeBERTa models). With the BERT teacher model, MPDistil shows significant AMargin of +1.5% on average compared to two most competitive base- lines, TinyBERT (average Maargin -0.5%) and MetaDistil (average AMargin = -1.0%). Even on the SuperGLUE test split, the BERT student model distilled with MPDistil out- performs the BERT teacher model on three out of six tasks (Table 5 in Appendix B.4). The margin between the teacher and student models is higher with the DeBERTa model. On COPA and RTE tasks, the DeBERTa--2-xxlarg model achieves nearly state-of-the art results, whereas the 12-layered student model achieves significantly lower performances. However, the average per- formance drop with MPDist il1 distillation is 4.6%, significantly lower than the other distillation methods CKD (9.8%) and MetaDistil (6%). MPDistíl shows similar competitive performance on GLUE dev set (Table 4 BERT-6L distilled with MPDistil achieves an average improvement of 3.1% over the base student model. Similar improvements are observed on the GLUE test set (Table 7n in Appendix B.4). In terms of accuracy, the student model distilled with MPDist i out- performs the teacher BERT-base model on four GLUE tasks on both the dev and test split. Table 6 and Table g in Appendix B.4 show that MPDist.f achieves the highest Aarggin among all the KD techniques on four tasks on both GLUE dev and test datasets.
Mobill DEkriin que
2000
-488
-24
-0.7
-14.0
-23
-9.1
SiDllgullr ar 2020
-10.1
-17.3
-1.0
-14.8
-9.0
-128
Minir Warker pa2rh
-3.
-119
-..0
-5.3
-1.2
-14.4
MiniL
-14.3
LRC Frecea
.664
-112
Annealing Krbariinnn mi
-88
-59
.1400
.112
CKD et ai 2011
-78
-66
-1.0
-11.7
-73
-11.2
Univeran KrAWEr
1001a]
-188
-54
-73
-28
-06
-11.2
DirroWWat Wack 60210
-39
-99
6.0
-7.
-54
8.6
Comionraaexi 11
al.
3023
-80
-7.1
27
-142
.79
-13.1
RAIL- KiLL-KKHHddree
-124
MGSKD
-128
MetaDisil 2500 404
0.9
MPDE a (Clis)
-19
7.0
04
1.00
(-) Cariculum learning
-28
-33
4.0
-18
122
0.0
Table 2: AMargin reported for different distillation meth- ods on SuperGL.UE tasks dev split with BERT models. The higher the value of AMargin, the more effective the distillation technique. AMargin > 0 indicates that the student outperforms the teacher. The results highlighted with are taken from Tan et al. (2023). For MPDi st stil and the ablation, we use the best scores obtained across different loss and reward configurations.
7

---

Published as a conference paper at ICLR 2024
ONLI
QQP
AcoF1
Acc
PeanSSparr
Teacher (BERT base)
83/7833
$1.088.2
91.0
90.5177.3
67.1
89.7
89,1899.1
Studast (BERT-bese G1.1
77,87883
75,8815
85.0
89 1765.0
64.6
87.7
Distille Student
80M8008
50,00662
2 87.
202286
66.4
90.5
89 L8888
56.3
Metii Mott-teashhrr
83.8088.1
91.1
90.5187.6
67.8
89.2899.1
Clr
$4.1/88.2
Binary roward
80.5
786.7
87.6
90.3/756.
67.1
(+) Col loss Real reward
80.4/80.2
$1.6/86.8
87.7
90.3187.0
06.4
90.9
$9.1/88.9
56.3
(4) Coen loss Bitary soward
80 R1-490077
$1.6/86.8
87.6
90.3287.0
67.-5
91.4
82.1/88.9 9
(+) Com loss Real reward
60.3881D
$1.5/86. 8
87.7
90 90.3887.0
66.4
90.9
89. 1/88. 9
56.3
()) Carriculum learning Cal loes
79.728
82.0887.
88.3
90 21.2887.0 0
65.3
90.8
80: 1/88. 8
56.3
Curricuuum learning Com loss
79,8/810
$1.6/87.1
88.3
90.2796.8
65.3
89.1/88.8
Table 4: Results of our method DUPDistin 1) on the GLUE dev set with BERT model.
meato Lnn.
2.
Taokk
099
Colla
Compett ve
PR0
501
Binarr Rewwadd
Real Berraadd
Figure 2: Meta-teacher validation loss w.r.t. dif-
ferent meta-teacher loss formulations.
Figure 3: Student rewards for different tasks.
Methods
BoolQ
CB
COPA
RTE
WIC
WSC
We evaluate MPDisti with OPT-
PKD
1.3B (Zhang et al. 2022). decoder-
CKD 201
-68
-7.3
-27.0
-13.0
-0.2
-48
MetaDism Z500
2021
-166
-89
-13,0
-10.8
-1.1
2.3
only LLM. reported in Appendix [B.]]
KPDIsL (Ours)
4.1
-15.0
-9.7
Table 12 The average AMargin on Su-
(-) Curriculum learning
-12
-7.1 -17.0
-10.1
3.6
1.0
perGLUE tasks with MPDistil is -0.58,
Table 3: AMargin for different distillation methods on
significantly lower than PKD which
the dev split of SuperGLUE tasks with DeBERTa models.
achieves margin of -1.92(c.f. Ta-
ble [3). We report the standard deviation of performances obtained each episode of curriculum learning in Tables and [T in Appendix [.44 With BERT, the average standard deviation across all SuperGLUE and GLUE tasks remain meagre at 2.4% and 0.71%. %. respectively, indicating our framework's stability.
5 DISCUSSION
Can better learners teach better? Figure highlights the distribution of meta-teacher validation loss, and shows lower validation loss with collaborative loss most tasks than competitive loss. As Proposition suggested, lower collaborative loss than competitive loss asserts stronger students. We analyze how a lower loss enables a better meta-teacher and encourages a better student. Towards this, we observe the improvement shown by the meta-teacher Similarly, we calculate the improvements the distilled student shows over the fine-tuned student model. The Pearson's correlation coefficient between meta-teacher improvement and student im- provement under collaborative loss is 0.65 (p-value=5e- - 8); in contrast, the same under competitive loss is only 0.02 with insignificant p-value of 0.86. Overall, the Pearson's correlation coefficient between the meta-teat veta-teacher and student improvement values remains 0.40 (p-value=1e 5). With such a low p-value and strong positive correlation, we assert that a stronger meta-teacher leads to a better and more competitive student.
the original fine-tuned teacher.
How does higher student reward impact student's performance? Figure highlights the dis- tribution of rewards gathered by the distilled student across tasks. On the classification tasks, the expected binary reward obtained 0.81, as opposed to the expected real reward of 0.35. As per Proposition 2, the expected binary reward on these classification tasks is always higher than the ex- pected real reward. With binary reward formulation, the correlation between the performances of the distilled student and the meta-teacher is 0.77, significantly higher than the correlation with the real reward (0.60). On BoolQ, the expected binary reward is < 0.5, indicating that the student model more underconfident than the teacher model. This propagates to the poorer dev performance by
8

---

Published as a conference paper at ICLR 2024
the distilled student model. In contrast, the tasks with the highest expected reward such RTE, SST-2, CB and COPA, the distilled student outperforms the teacher model with the widest margins.
How does student select learning curricula? We analyze the cumulative weightage of different tasks in the student curriculum in Figure 4.. On the low-performing tasks such MNLI, MNLI-mm, and QNLI, the model explores all the different tasks with nearly uni- form weightage. Compared to this, on high-performing tasks such as RTE, CB, and COPA, the model perfectly balances exploration and exploitation in selecting the suitable curricu- lum. For instance, on CB, the model only chooses the COPA task in its curriculum. On the other hand, on COPA, CB has the highest weightage in the curriculum. On the RTE task, the model selects the same task repeatedly in the cur- riculum. To understand how the model preserves the curriculum throughout its learning, we calculate
(a)
in Lumcouum
a similarity metric, sim,,,+11 = C.sl be-
[Cyl tween two learned curricula C; and C++1 in two con-
secutive episodes. LCS denotes the longest common subsequence. We visualize the simm;,+41 distribu- tion for different tasks in Figure Ap. high sim- ilarity score indicates that the model only exploits and shows reluctance to learn new curricula. In-
MNU
S5T-
MALI-M
STS4
COPA
WSC
MAPC
BeotO
(b)
terestingly, the model has lower average similarity on the gegehperforming tasks, indicating more ex- ploration. However, after learning the most suitable curriculum, the model stops exploration and exploits the same set of tasks to preserve high rewards. On the other hand, on low-performing tasks like MNLI, MNLI-mm and QNLI, the model shows higher av- erage similarity with low variance, indicating more restrictive behavior.
Sintarty botweer curicula
MNL
ONU
30P
ATC
ST5
WNU
Bool
MKL
Chiqqure dicanne betovepn curiua
We further calculate the count of each task in each curriculum and the chi-square distance between con- secutive curricula understand how different tasks are preserved within curricula. We report the dis- tribution of chi-square distances in Figure 4c. For nigh-performing tasks, we observe significantly lower chi-square distance between consecutive cur- ricula. Lower chi-square and low average sim val- ues conclude that for these tasks, the curriculum learning model does not explore newer tasks in the curriculum but rather explores the different se- quences among them. It justifies that not only are the tasks important within curricula, but their sequence also equally matters.
MNU
WNLLMWW
MSPC
ONU
007
ATE
5552
WTC
wiic
Figure 4: (a) Cumulative weightage differ- ent tasks in final student curriculum for Su- perGLUE and GLUE benchmarks. tribution of curriculum similarities for differ- ent SuperGLUE and GLUE tasks. High cur- riculum similarity indicates that the student model preserves the curriculum in consecu- tive iterations. (c) Distribution of Chi-square distance between consecutive curricula.
(b) Dis-
CONCLUSION
This paper introduced MPDi rrDistii ittill 1, method for distilling meta-policy knowledge, emphasizing the importance of creating suitable joint utility to enhance the teacher's distillation capabilities. Fur- thermore, underscored the necessity of employing multi-task curricula to produce more robust and effective student models through teacher knowledge distillation. framework is generic and be applied distilling large language models, regardless of their underlying architecture. While our empirical study primarily focused on natural language under- standing tasks, our methodology can also be extended other tasks involving natural language generation and reasoning. While our student curricula were designed with tasks featuring similar abstractions, would be worth noting the potential impact of diverse tasks with varying levels of abstraction and complexity on enhancing the generalizability of student models.
essential to note that our

---

Published as a conference paper at ICLR 2024
REFERENCES
Gustavo Aguilar, Yuan Ling, Yu Zhang, Benjamin Yao, Xing Fan, and Chenlei Guo. Knowledge distillation from internal representations. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 34, pp. 7350-7357, 2020.
Martin Arjovsky, Soumith Chintala, and Léon Bottou. Wasserstein generative adversarial networks. In International conference on machine learning, pp. 214-223. PMLR, 2017.
Yoshua Bengio, Jérôme Louradour, Ronan Collobert, and Jason Weston. Curriculum learning. In Proceedings of the 26th annual international conference on machine learning, pp. 41-48, 2009.
Luisa Bentivogli, Peter Clark, Ido Dagan, and Danilo Giampiccolo. The fifth pascal recognizing textual entailment challenge. TAC, 7:8, 2009.
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Advances in neural information processing systems, 33:1877-1901, 2020.
Daniel Cer, Mona Diab, Eneko Agirre, Inigo Lopez-Gazpio, and Lucia Specia. Semeval-2017 task
I: Semantic textual similarity mumilartt-multllinuuu and loosull focused evaluation. arXiv preprint arXiv:1708.0000 00055, 2017.
Zihan Chen, Hongbo Zhang, Xiaoji Zhang, and Leqi Zhao. Quora question pairs, 2018.
Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, et al. Palm:
Scaling language modeling with pathways. arXiv preprint arxiv:2204.023// 02311, 2022.
Christopher Clark, Kenton Lee, Ming-War Wei Chang, Tom Kwiatkowski. Michael Collins, and Kristina Toutanova. Boolq: Exploring the surprising difficulty of natural yes/no questions. arXiv preprint arXiv: /905. 10044, 2019.
Ido Dagan, Oren Glickman, and Bernardo Magnini. The pascal recognising textual entailment challenge. In Machine learning challenges workshop, pp. 177-190. Springer, 2005.
Marie-Catherine De Mamneffe, Mandy Simons, and Judith Tonhauser. The commitmentbank: In- vestigating projection in naturally occurring discourse. In proceedings of Sinn und Bedeutung. volume 23, pp. 107-124, 2019.
Lei Deng, Guoqi Li, Song Han, Luping Shi, and Yuan Xie. Model compression and hardware acceleration for neural networks: A comprehensive survey. Proceedings of the IEEE, 108(4): 485-532, 2020.
Jacob Devlin, Ming- Wiig-Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint 1.Xi:18100004000 2018.
Bill Dolan and Chris Brockett. Automatically constructing corpus of sentential paraphrases. In Third International Workshop Puraphrasing (IWP2005), 2005.
Hao Fu, Shaojun Zhou, Qihong Yang. Junjie Tang, Guiquan Liu, Kaikui Liu, and Xiaolong Li. Lre- bert: latent-representation contrastive knowledge distillation for natural language understanding. In Proceedings of the AAAI Conference Artificial Intelligence, volume 35, pp. 12830-12838, 2021.
Danilo Giampiccolo, Bernardo Magnini, Ido Dagan, and William Dolan. The third pascal rec- ognizing textual entailment challenge. In Proceedings of the ACL-PASCAL workshop textual entailment and paraphrasing, pp. 1-9, 2007.
Yuxian Gu, Li Dong, Furu Wei, and Minlie Huang. Knowledge distillation of large language models. arXiv preprint arXiv:2306.08555 2023.
Md Akmal Haidar, Nithin Anchuri, Mehdi Rezagholizadeh, Abbas Ghaddar, Philippe Langlais, and Pascal Poupart. Rail-kd: Random intermediate layer mapping for knowledge distillation. arXiv preprint arXiv: 10164, 2021.
10

---

Published as a conference paper at ICLR 2024
R Bar Haim, Ido Dagan, Bill Dolan, Lisa Ferro, Danilo Giampiccolo, Bernardo Magnini, and Idan Szpektor. The second pascal recognising textual entailment challenge. In Proceedings of the Second PASCAL Challenges Workshop on Recognising Textual Entailment, volume pp. 785- 794, 2006.
Pengcheng He, Xiaodong Liu, Jianfeng Gao, and Weizhu Chen. Deberta: Recoding-enhancee bert with disentangled attention. arXiv preprint arXiv:2006.03654 2020.
Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. Distilling the knowledge in neural network. arXiv preprint arXiv: 1033.0530330233 2015.
Aref Jafari, Mehdi Rezagholizadeh, Pranav Sharma, and Ali Ghodsi. Annealing knowledge distilla- tion. arXiv preprint arXiv:2104.07163, 2021.
Aref Jafari, Ivan Kobyzev, Mehdi Rezagholizadeh, Pascal Poupart, and Ali Ghodsi. Continuation kd: Improved knowledge distillation through the lens continuation optimization. arXiv preprint arXiv:2212.05999. 2022.
Xiaoqi Jiao, Yichun Yin, Lifeng Shang, Xin Jiang, Xiao Chen, Linlin Li, Fang Wang, and Qun Liu. Tinybert: Distilling bert for natural language understanding. arXiv preprint arXiv:1909.100 10351, 2019.
Xiao Jin, Baoyun Peng, Yichao Wu, Yu Liu, Jiaheng Liu, Ding Liang, Junjie Yan, and Xiaolin Hu. Knowledge distillation via route constrained optimization. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 1345-1354, 2019.
Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. Scaling laws for neural language models. arXiv preprint arXiv:2001.0836 2020.
Taehyeon Kim, Jaehoon Oh, Nik Kim, Sangwook Cho, and Se- Se-Young Yun. Comparing kullback-leibler divergence and mean squared error loss in knowledge distillation. arXiv preprint arXiv:2105.08919, 2021.
Jannik Kossen, Tom Rainforth, and Yarin Gal. In-context learning in large language models learns label relationships but is not conventional learning. arXiv preprint arXiv:2307. 12375, 2023.
Hector Levesque, Emest Davis, and Leora Morgenstern. The winograd schema challenge. In Thir- teenth international conference on the principles of knowledge representation and reasoning, 2012.
Zheng Li, Xiang Li, Lingfeng Yang, Borui Zhao, Renjie Song, Lei Luo, Jun Li, and Jian Yang. Curriculum temperature for knowledge distillation. In Proceedings of the AAAI Conference Artificial Intelligence, volume 37, pp. 1504-1512, 2023.
Chang Liu, Chongyang Tao, Jiazhan Feng, and Dongyan Zhao. Multi-granularity structural knowl- edge distillation for language model compression. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. pp. 1001-1011, 2022a.
Jihao Liu, Boxiao Liu, Hongsheng Li, and Yu Liu. Meta knowledge distillation. arXiv preprint arXiv:2202.079900. 2022b.
Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. arXiv preprint 1rX1v:1711.51011, 2017.
Seyed Iman Mirzadeh, Mehrdad Farajtabar, Ang Li, Nir Levine, Akihiro Matsukawa, and Hassan Ghasemzadeh. Improved knowledge distillation via teacher assistant. In Proceedings of the AAAI conference on artificial intelligence, volume 34, pp. 5191-5198, 2020.
Haojie Pan, Chengyu Wang, Minghui Qiu, Yichang Zhang, Yaliang Li, and Jun Huang. Meta-kd: A meta knowledge distillation framework for language model compression across domains. arXiv preprint ar arXiv:20/2.01266 2020.
11

---

Published as a conference paper at ICLR 2024
Geondo Park, Gyeongman Kim, and Eunho Yang. Distilling linguistic context for language model compression. arXiv preprint arXiv:2109.0883 2021.
Peyman Passban, Yimeng Wu, Mehdi Rezagholizadeh, and Qun Liu. Alp-kd: Attention-based layer projection for knowledge distillation. In Proceedings of the AAAI Conference on artificial intelligence, volume 35, pp. 13657-13665, 2021.
Mohammad Taher Pilehvar and Jose Camacho-Collados Wic: the word-in-context dataset for eval- uating context- meaning representations. arXiv preprint arXivv/808.09122 2018.
Melissa Roemmele, Cosmin Adrian Bejan, and Andrew S Gordon. Choice of plausible alternatives: An evaluation of commonsense causal reasoning. In 20/1 AAAI Spring Symposium Series, 2011.
Victor Sanh, Lysandre Debut, Julien Chaumond, and Thomas Wolf. Distilbert, a distilled version of bert: smaller, faster, cheaper and lighter. arXiv preprint arXiv: 2019.
Wenxian Shi, Yuxuan Song, Hao Zhou, Bohan Li, and Lei Li. Learning from deep model via exploring local targets. 2020.
Richard Socher, Alex Perelygin, Jean Wu, Jason Chuang, Christopher D Manning, Andrew Ng, and Christopher Potts. Recursive deep models for semantic compositionality over sentiment treebank. In Proceedings of the 2013 conference on empirical methods in natural language pro- cessing, pp. 1631-1642, 2013.
Siqi Sun, Yu Cheng, Zhe Gan, and Jingjing Liu. Patient knowledge distillation for bert model compression. arXiv preprint arXiv: 1908..0.0000000 09355, 2019.
Zhiqing Sun, Hongkun Yu, Xiaodan Song, Renjie Liu, Yiming Yang, and Denny Zhou Mobile- bert: a compact task-agnostic bert for lisitee devices. arXiv preprint arXiv:2004.02000 02994, 2020.
Richard Sutton, David McAllester, Satinder Singh, and Yishay Mansour. Policy gradient meth- ods for reinforcement learning with function approximation. Advances in neural information processing systems, 12, 1999.
Shicheng Tan, Weng Lam Tam, Yuanchun Wang, Wenwen Gong, Yang Yang, Hongyin Tang. Keqing He, Jiahao Liu, Jingang Wang, Shu Zhao, et al. Gkd: A general knowledge distillation framework for large-scale pre-trained language model. arXiv preprint arXiv:2306. 06629, 2023.
Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302.13977 2023.
Iulia Turc, Ming- Wei Chang, Kenton Lee, and Kristina Toutanova. Well-read students learn better: On the importance of pre-training compact models. arXiv preprint arXiv: NXNNv:/908.09000 2019.
Cédric Villani et al. Optimal transport: old and new, volume 338. Springer, 2009.
Alex Wang. Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R Bowman. Glue: A multi-task benchmark and analysis platform for natural language understanding. arXiv preprint arXiv:1804.0746)) 2018.
Alex Wang, Yada Pruksachatkun. Nikita Nangia, Amanpreet AmanpreetSSinhh Singh, Julian Michael, Felix Hill, Omer Levy. and Samuel Bowman. Superglue: A stickier benchmark for general-purpose language understanding systems. Advances in neural information processing systems, 32, 2019.
Wenhui Wang, Hangbo Bao, Shaohan Huang, Li Dong, and Furu Wei. Minilmv2: Multi- head self-attention relation distillation for compressing pretrained transformers. arXiv preprint arXiv:2012.15828, 2020a.
Wenhui Wang, Furu Wei, Li Dong. Hangbo Bao, Nan Yang, and Ming Zhou. Minilm: attention distillation for aank-tgnottic compression of pre-trained transformers. Advances in Neu- ral Information P Processing Systems, 33:5776-5788. 2020b.
Deep self-
12

---

Published as a conference paper at ICLR 2024
Adina Williams, Nikita Nangia, and Samuel R Bowman. A broad-coverage challenge corpus for sentence understanding through inference. arXiv preprint arxbv:1004.0522, 2017.
Ronald J Williams. Simple statistical gradient- following algorithms for connectionist reinforcement learning. Machine learning, 8:229-256, 1992.
Meng-Chieh Wu and Ching-Te Chiu. Multi-teacher knowledge distillation for compressed video action recognition based on deep learning. Journal of Systems Architecture, 103:101695, 2020.
Yimeng Wu, Mehdi Rezagholizadch, Abbas Ghaddar, Md Akmal Haidar, and Ali Ghodsi. Universal- kd: Attention-basee output- output-groundt grounded intermediate layer knowledge distillation. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pp. 1669, 2021a.
Zhengxuan Wu, Atticus Geiger, Josh Rozner, Elisa Kreiss, Hanson Lu, Thomas Icard, Christo- pher Potts, and Noah D Goodman. Causal distillation for language models. arXiv preprint arXiv:2112.02505. 2021b.
Liuyu Xiang, Guiguang Ding, and Jungong Han. Learning from multiple experts: Self- knowl- edge distillation for long-tailed classification. In Computer Vision-ECCV 2020: 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part V 16, pp. 247-263. Springer, 2020.
Canwen Xu, Wangchunshu Zhou, Tao Ge, Furu Wei, and Ming Zhou. of Compressing bert by progressive module replacing. arXiv preprint arXiv:2002.02923, 2020.
Ze Yang, Linjun Shou, Ming Gong, Wutao Lin, and Daxin Jiang. Model compression with two- stage multi-teacher knowledge distillation for web question answering system. In Proceedings of the 13th International Conference on Web Search and Data Mining, pp. 690-698, 2020.
Fei Yuan, Linjun Shou, Jian Pei, Wutao Lin, Ming Gong, Yan Fu, and Daxin Jiang. Reinforced multi-teacher selection for knowledge distillation. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 35, pp. 14284-14291, 2021.
Susan Zhang, Stephen Roller, Naman Goyal, Mikel Artetxe, Moya Chen, Shuohui Chen, Christo- pher Dewan, Mona Diab, Xian Li, Xi Victoria Lin, et al. Opt: Open pre-trained transformer language models. arXiv preprint arXiv:2205.01068, 2022.
Ying Zhang, Tao Xiang, Timothy M Hospedales, and Huchuan Lu. Deep mutual learning. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 4320-4328, 2018.
Wangchunshu Zhou, Canwen Xu, and Julian McAuley. Bert learns to teach: Knowledge distillation with meta learning, arXiv preprint arXiv:2106.044 2021.
Xunyu Zhu, Jian Li, Yong Liu, Can Ma, and Weiping Wang. A survey on model compression for large language models. arXiv preprint arXiv:2308.07633, 2023.
A PROOFS OF THE THEORETICAL RESULTS
A. PROOF OF PROPOSITION
The expected collaborative and competitive loss for any given teacher and student logit is defined as E log D(.T)) log )). and E log (i.r) + log ()),) respectively.
13

---

Published as a conference paper at ICLR 2024
c meta col
[ meta com
B
log ((rT)
log ((i
2og B(r,) + log (i.
Blog ((,rr))
2oo ((sss))
Blog ((i,Ty)
Ellog V(r,5)]
Bllog V(s.s)
log ((,)))) > 0
log Ely
log B.yr)) (Jensen's inequality)
E
log (()
E ( 1.
E ((,))
As D(i,s) > Ky.sy. it proves that ().)) > E[jrr))
A.2 PROOF OF PROPOSITION 2
B[ = P Pu..s) ((c.s) - MirT' > 0
[ dìcs,s) diarT"
Being two probability distributions, the random variable ((s.s) U(r)') has its range [-1, 1].
E(s(r.)) - f
011,5 -0 l (Dc..) - mirrr)) dis.ss) dūa,r))
[r Bi..) BUTTD
(Jc.s) - MGrrr) dics,s) dū(ar"")
(B1..) MO.T)) deliss) di(a."))
The second term in the above expression is less than or equal Ú(i.s) Our'' S for any ((,s)) [0, ). Therefore.
[(o.ss) - (urT) S
(ii1.s) -(aTr) div.s) digry
; Bur's B-Br
dge,s) dijrrr) (Change of limits)
(o..9) Purr' se
diuuss dū
Jis.st "o,r' =
E( ) J.
B EXPErIMENTAL DETAILS AND RESULTS
B. DATASETS
This section briefly describes the tasks of SuperGLUE (Wang et al. 2019 and GLUE (Wang et al. 2018) benchmarks.
14

---

Published as a conference paper at ICLR 2024
B.1.1 SUPERGLUE BENCHMARK
BoolQ: Boolean Questions comprise of binary questions using the Google search engine as their source of questions; they are then paired with appropriate paragraphs from Wikipedia articles that contain the relevant answers (Clark et al. 2019).
CB: CommitmentBank comprises of short texts with embedded clauses. The examples are taken from sources like British National Corpus Fiction and Wall Street Journal. It involves a three-class textual entailment task. Each example includes premise and the corresponding hypothesis along with the class label "contradiction", "neutral", or "entailment" (De Marneffe et al., 2019).
COPA: Choice of Plausible Alternatives is causal reasoning task which involves selecting the most plausible choice for a cause or effect given a premise. (Roemmele et al.) 2011).
WiC: Word-in -entteett is task focused on word sense disambiguation, comprising of binary clas- sification pairs of sentences. In this task, two text snippets are provided, each containing a word that could have multiple meanings. The goal ascertain whether the word has the same meaning in both sentences (Pilehvar & Haimacho-Colladde 2018).
WSC: Winograd Schema Challenge is coreference resolution task where each sentence example includes a passage, a pronoun, and a choice of nouns from that sentence. The task objective is to determine whether the pronoun and noun refer each other or not (Levesque et al., 2012).
B.1.2 GLUE BENCHMARK
SST-2:
Stanford Sentiment Treebank is a dataset derived from movie reviews, annotated with senti- ment labels. The goal is predict the sentiment "rositive"""negative" of the reviews, employing two-way classification Socher et al. (2013).
Microsoft Research Paraphrase Corpus consists of sentence pairs obtained from online news sources and are buman-annotater to ascertain their semantic equivalence. This task involves predicting whether two given sentences are paraphrases of each other or not (Dolan & Brockett) 2005).
MRPC:
STS-B: Semantic Textual Similarity Benchmark predicts the degree of similarity between two given sentences on scale from 0-5. The dataset is compiled from diverse origins, including videos and image descriptions, news headlines, and natural language inference information. (Cer al. 2017).
QQP: Quora Question Pairs involves predicting whether two given questions asked on the question- answering website Quora are semantically equivalent or not (Chen et al. 2018).
Multi-Genre Natural Language Inference involves predicting whether given hypothesis is "entailed", "neutral", or "contradicted" for given premise. The dataset is collected from various sources, including fiction, government reports, and transcribed speech (Williams et al. 2017).
MNLI:
QNLI: Question-answering NIII task based on the Stanford Question Answering Dataset. involves determining whether given context sentence contains the answer to corresponding ques- tion Wang et al. (0018).
RTE: Recognizing Textual Entailment is compiled from collection of datasets that originate from series of annual textual entailment challenges (Dagan et al., 2005 Haim et al., 2006; Giampiccolo et al. 2007: Benttvvgll et al., 2009). These datasets are constructed using news and Wikipedia text sources. The datasets are transformed into a two-class split, collapsing "neutral" and "contradiction" categories into "not entailment".
WNLI: Winograd NLI requires selecting the referent of a pronoun from choices by converting into sentence pair classification. The task assesses whether substituting the pronoun with possible referents results in sentence entailment (Levesque et al. 2012).
B.2 EXPERIMENTAL SETUP
For all the SuperGLUE and GLUE tasks, maximum sequence length of 128. All the results reported in the paper (except for the results obtained from other basclines) are obtained from running grid search over the of hyperparameters, teacher learning rate from (2e-5, 3e-5), the student
15

---

Published as a conference paper at ICLR 2024
learning rate from (2e-5, 3e-5), from {4.0, 5.0, 6.0, 7.0), from (0.4, 0.5, 0.6), and / from {80, 90, 100). The discount factor ? is set as 0.99. The meta-teacher and the curriculum models are trained with fixed learning rate of 0.001. We train all the models with maximum of 10 epochs, and the curriculum model is trained with 200 episodes. We set the training, quiz and validation hatch size 8. All the models trained with Adam optimizer with weight decay (Loshchilov & Hutter 2017). One Tesla V100 and A100-40 GPU were used for conducting the experiments.
B.3 BASELINES
To evaluate the impact of our proposed distillation method, we undertake several competitive KD and MetaKD baselines.
PKD (Sun et al. 2019) uses a patient learning approach where instead of using the last layer of teacher for distilling knowledge into the student, the student utilises the intermediate layer represen- tation of the teacher model.
CKD Park al. (021)) transfers knowledge from teacher to student, utilising the word representa- tions contextual knowledge. It defines two contextual knowledge objectives Word Relation (WR) and Layer Transforming Relation (L.T),, examining the relationships among words in particular layer and across different layers, respectively.
TinyBERT (iao al. 2019) employs two-stage framework, general and task- distillation of BERT-base into a tiny 4ayyr student model, using a multiple distillation objective. addition to logit-based matching. the teacher transfers its knowledge through embedding layers, intermediate hidden states, and attention matrices.
Zhou al. (2021) argued that the teacher model is not optimized for transferring knowl- edge to the student model. MetaDistil proposes trainable teacher setup wherein the student's pre- dictive performance is considered while fine-tuning the teacher model using quiz dataset. The updated teacher is further used to distil knowledge into the student further.
MetaDistil:
B.4 RESULTS
Table Table highlight different model performances on SuperGLUE and GLUE tes sets, re- spectively. BERT-6L distilled with MPDistiD i1 achieves better accuracy than the BERT-base teacher model in three SuperGLUE and four GLUE tasks. Table 66 and Table 8 highlight the Marggi for MPDistil 1 GLUE dev and test respectively. It worth noting that most contemporary KD methods fail to improve the student's performance over the teacher's on GLUE test. In contrast, the student model distilled with MPDisti il achieves positive Marggn on MRPC, QQP, RTE and SST-2. The ablation model without curriculum learning achieves positive AMargin on the STS-B task. These superior performances demonstrate the generalizability of the student model distilled with MPDistil.
Methods
BoolQ
CB
COPA
RTE
WiC
WSC
Teacher (BERT-hase)
74.6
83.6
56.6
63.2
54.8
65.1
Student (BERT-base 6L.
71.3
81.2
53.0
64.1
56.4
65.1
Distilled Student
72.1
82.0
52.8
64.3
53,9
65.1
Meta-leachier
(+) Col
73.0
$6.8
$4.9
65.1
(+) Com loss
72.9
84.0
58.8
62.9
54.4
64.4
Student with MPBT
(+) Col loss Binary reward
72.1
81.2
50.6
64.2
$3.8
63.0
(+) Col loss Real reward
71.1
82.4
53.6
64.2
53.8
64.4
(+) Com loss + Binary reward
71.2
82.0
51.6
64.5
55.0
65.8
(+) Currieulum learning Col loss
Real reward
81.6
54.7
72.4
82.0
53.0
62.8
$4.3
65.1
Curriculum learning Comp loss
72.7
82.4
53.0
64.2
55.2
65.8
Table 5: Results of method the SuperGLUE test sets with the BERT model.
Submissions
httpss:/super. www.ggggwwawe.k.ccom
and
https:///glubbechheee
com/
16

---

Published as a conference paper at ICLR 2024
Methods
MNLI/MNII-M
QNLI
QQP
RTE
SS1-2
STS-B
Acc
Acc/F1
Aco/F1
Pear/Spear
KD Hinton et al.
2015
-2.0-1177
-2.1/2..0
-2.0
-0.5/1122
-3.7
-1.8
-.66--166
PKD
-1.9/-1.6
-2.9/2222
-1.7
+0.//00.7
-3.8
1.66/1.7
TinyBKKI
2019
-1.0/1111
-1.1/1111
-1.4
-0.8/-0.9
-3.7
-1.1
1.001111
RCO Jin al. 2019
-2.2/-2.0
-2.5/-2.1
-1.5
-0.8/-111
-3.8
-1,6
-1,5/1155
TAKD Miradeh
2020
-2.1/-199
-2.6/2.00
-1.6
-0.7/-1.0
-2.9
-1.6
-2.0/-188
DML Zhang el al. 201
-2.2/-2.0
-2.5/-2.0
-1.6
-1.1/-1.1
-3.0
-1.5
-1.8/-1.7
ProK Shi al
2020
-18//1.77
-1.3/-0.9
-1.5
0..//006
-3.0
-1.7
-1.3/1122
SFTN
al
-2.2/22.0
-2.3/-18
-1.0/-1.0
-29
-1.5
-1,8/-13
Metal Zhou
20211
-1.1/-111
-0.8/-0.5
-0.8
+0.4-0.4
-2.0
-0.7
-0.8/-0.7
MPDist st (Ours)
-3.2/-2.3
-2,41-14
-3.3
-0.2/-0.3
0.4
1.1
-0,3/-0,2
(-) Curriculum leamning
-4.0/-2.4
-2.0/-100
-2.6
0.3/-0.3
-1.8
1.3
-0.4/-0.3
Table 6: AMargin reported for different distillation methods on GLUE tasks dev split with the BERT model. The results highlighted with i are obtained from Zhou et al. (202I).
Methods
MNLIMNLI-MM
MRPC
QNLI
QOP
RTE
SST-2
STS-B
WNLI
Acc
Acc/F
Acc
AccFT
Ace
Acc
Pear/Spear
Acc
base)
87.9/70.3
Student (BERT-base 61.)
79.5/78.2
80.5/86.0
86.9
87. 87.9/68.6
64.1
90.1
83.8/82.5
65.1
Distilled Student
80.0/79.6
79.6/85.5
86.7
1/69.
83.1/81.8
65.1
Meta-teacher
(+) loss
84.1/83.0
81,4/86.7
90.3
87.9/70.2
62.8
92.5
83.3/82.0
65.1
Com loss
84.0/K3
81,6/86.8
90.2
88.2/70.4
62.9
83.3/82.0
65.1
Student with MPDiatil
Binary
80.6779.6
81,4/86.8
86.7
88.1/69.3
83.0/81.7
$8.9
(+) Cul luss Real reward
80.2/78.5
81.6/86.8
86.8
88.2/69.4
64.2
91.2
83.2/%
65.1
(+)
Binary reward
86.9
88.2/69.3
(4) Com loss Real reward
80.2/80.1
81.5/86.8
86.8
88, 28.2/69.4
64.2
91.2
83,2/8
65.1
learning
79.8/79.9
84,2/83.1
Curriculum learning Com loss
79.4/79.9
81.6/87.1
88.2
88.5/70.1
64.2
91.0
1/83.1
65.1
Table 7: Results of our method (MPDistil) on the GLUE test set with the BERT model.
Methods
MNLI/MNLI-MM
MRPC
QNLI Acc
QQP Acc/F1
KTE
SST-2
STS-B
Acc
Acc/F1
Acc
Acc
Pear/Spear
KD
Turc et al.
2019
-181112
-3.1/2.11
-1.6
-0,3/-08
-1.1
-1.7
PKD Sun el
2019
331/2244
4.9/33.9
-1.5
-0.3/0.5
-0.9
-1.5
-3,7/4.22
BERT-ol heseus
al.
2020
222//11.
-1.6/-1.3
-0,9
0.1/0.4
-0.2
-1.5/-1.7
ProKT
Shi al.
2020
-171-122
-2.5/-1.9
-0.8
-0,3/-033
-0.2
TinyBERI Jua
2019
-1.6/0.88
-2.0/-1.0
-0.7
-0.6/0.3
0.4
-0.4
-1,3/112
DML
Zhang al 2018
-2.0/11.8
-3.6/-2.4
-1.0
-0,5/-0,5
-0.1
-0.8
-1,6/-188
RCO
Jin al. 9019
-2.112222
-3.4/-211
-1.2
-0.5/-088
0.1
-0.9
-1.8/-177
TAKD
Mirzadeh
2020
-22//11.7
-3.51-2.4
-1.1
-0.4-066
0.4
-0.6
-17711.7
SFIN
Park
20/1
-25112111
-3.6/-2.4
-088-1.0
-2.0-1.6
MetaDistil Zhou 202
-0.8/-0.2
-0.1/-0.2
-0.3/-.11
-1.0/-.88
MPDist (Ours)
-3.3/-2.9
0.3/-0.1
-3.4
0.3/-0.9
1.3
-0.6
-0.4/0.4
(-) Curriculum learning
-..1/2.9
0.7/0.4
-2.0
0.6/0.1
1.0
-0.4
0.7/1.0
Table 8: Margin reported for different distillation methods GLUE tasks test split with the BERT model. The results highlighted with i are obtained from Zhou et al. (202I)
Methods
BoolQ
COPA
RTE
Student with MPDistil
(+) Col loss
Binary reward
72.8
1:
0.35
3.31
4.13
67.1
1
58.0 1
1.05
65.4
5,36
Col loss Real reward
73.4
0.71
82.1
1.49
70.0
4.22
66.4 -
1.44
58.6
0.78
64.4
4.24
(+) Com loss Binary reward
73.0
0.63
80.4
1.55
62.0
1
3.80
67.5
1
1.37
59.6
1
1.63
65.4
4.12
(+) Com loss
Real reward
73.0 1
0.34
78.6
2.19
63.0
1
4.38
66.4
1
2.22
58.9
1
0.99
65.4
2.56
Table 9: Performance of our method (MPDistil) along with standard deviation the SuperGLUE dev sets with the BERT model.
17

---

Published as a conference paper at ICLR 2024
Methods
BoolQ
CB
CОРА
RTE
WiC
WSC
Student with MPDisti
Col loss Binary zeward
82.7
2.86
76.8
3.91
67.0
79.8
2.25
61.0
0.96
66.3
0.76
(+)
Col loss Real reward
82.7 + 0.18
76.7
1,31
67.0 + 2.87
78.0 - 1.08
60.0 = 0.53
65.4 = 4.40
(+) Com loss Binary reward
82.4 + 0.49
76.8
+
2.26
69.0
+ 3.90
80.9
+ 2.27
59.7
+ 0.47
65.4
2.94
(+) Com loss Real reward
82.6 + 0.22
76.8 +
1.27
68.0
+ 3.28
77.6 + 1.28
60.0
= 0.45
64.4 3.87
Table 10: Performance of our method (along with standard deviation) including performance vari-
the SuperGLUE dev with the DeBERTa model.
2.39
0.58
WNLI
+
1
56
56.3
56.3
0:39
0.72
0.20
0.60
Pear Spert/
0600899
020889
A88
STS-B
0.70
8911
89
89.
89.1
0.56
0-2
0.89
0.20
SSTE ACC
1
0000
91.4
009
2.00
3.74
3.96
3.73
WTT:
Avc
67. 1:
631
664
06.4
0.28 N6. 3 0.4
1.09
0.5 87700 1. 1.0
0.32
0.997700
0128700
O
Acoi
9022
98.3
9813
107
1.43
= 087
1.47
ONLO
ACC
=
877
N77
145 3 1.13
0.71
1.38 8688 1 1.53
0.88
8 +
01368868
MRRCC
SCOFT
816 1
81.1
0.70
173
034
061
MINLMMNNL--MD
2003
8012
N0.4 L 1:15 A0L7
1,22810
1
N0.4 + 40
801
(4) Col loss Money Roward
(t) Com loss Dinury rewaadd
(+) Cil lasa Redl couand
Studen wi Bittett
Real
Table 11: Performance of our method (MPDistil) along with standard deviation the GLUE dev set with BERT model.
18

---

Published as a conference paper at ICLR 2024
Methods
BoolQ
CB
COPA
RTE
WIC
WSC
Teacher (OPT-1.3P 1.3B)
64.9
83.9
55.0
53.4
56.7
63.5
Student (OPT 12L)
63.4
75.0
54.0
53.1
53.8
63.5
Distilled Student
65.6
67.9
55.0
52.7
54.7
63.5
Meta- -eachee
(+) Col loss
64.9
80.4
55.0
50.9
56.
63.5
(+) Com loss
65.4
82.1
55.0
53.8
56.0
63.5
Student with MPDistiI
(+) Col loss Binary reward
65.7
71.4
55.0
55.6
54.7
63.5
Col loss + Real reward
65.6
73.2
55.0
54.9
56.1
63.5
Com loss Binary reward
65.6
76.8
55.0
53.8
56.1
63.5
(+) Com loss + Real reward
65.6
73.2
55.0
53.8
56.7
63.5
Curriculum learning Col loss
65.6
73.2
55.0
52.7
54.7
63.5
(-) Curriculum learning + Comp loss
64.3
71.4
54.0
54.5
54.9
63.5
Table 12: Performance of our method (MPDistil) on the SuperGLUE dev with OPT model.
Methods
BoolQ
CB
COPA
RTE
WiC
WSC
PKD Sun et al. 2019
-0.5
-8.9
0.0
0.7
-2.8
0.0
MPDistil (Ours)
0.8
-7.1
1.5
0.0
0.0
(-) Curriculum learning
0.7
-10.7
0.0
1.0
-1.8
0.0
Table 13: AMargin for different distillation methods on SuperGLUE dev tasks with OPT model.
19


Reviews:
[{"review_id": "ps2nkyGeN6", "reviewer": "Reviewer_AnAG", "summary": "This paper proposes a meta-policy distillation method MPDistill which consists of four steps. First, the teacher is fine-tuned on the task. second, the student is fine-tuned by using the task loss and the distillation loss. Third, the meta-teacher is trained to optimize the teacher's outputs and the student's representations. Fourth, a student curriculum model is trained to generate a sequence of tasks for student fine-tuning and rewards calculation. Experiments show promising results.", "questions": "1. Meta-teacher uses a bi-optimization, which is computationally expensive. What is the training cost of the proposed method compared to the baselines?\n2. The curriculum model parameters are learned via a reinforcement learning loss, which may lead to unstable training. The performance variance of the proposed method is not reported.\n\n-----------------------------------------------------\nThanks for the authors' response. I keep my rating.", "limitations": "", "rating": 6, "confidence": 1, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. Using a curriculum model to learn a set of tasks for the student is interesting.\n2. Comparison with different methods are reported.", "weaknesses": "1. The framework is very heavy since it consists of bi-level optimization and reinforcement learning loss.\n2. The framework consists of four steps. Some important training details are missing. It is hard to reproduce.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "HMrHJe6wcH", "reviewer": "Reviewer_JBkZ", "summary": "This paper studies the problem of meta-learning based knowledge distillation, where the teacher model should be tuned during distillation.\nThe proposed made several adjustment to this setting with teacher fine-tuning, meta-teacher learning and student curriculum learning, each of which can somehow improve the performance.\nThe authors conduct extensive experiments on the natural language understanding tasks and the results somehow demonstrate the effectiveness of the proposed method.", "questions": "* Please clearly organize the challenges and motivation of this paper.\n* Concerning the detailed design of each part, what are they purposed for?", "limitations": "", "rating": 5, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "* The meta-learning-based knowledge distillation task is interesting and still under-explored.\n* The experiments are extensive.", "weaknesses": "* My major concern is that the paper is overall hard to follow. Specifically, in the introduction section, the authors list several challenges of MetaKD, while these challenges seems scattered and the major challenges are missed.\n* The proposed method consists of several steps and the authors make several adjustment in each step. However, these adjustment seems quite straightforward, I wonder the purpose of these design and why it works.\n* Concerning the detailed design of the proposed method, the meta-teacher takes hidden state representation from both the teacher and student models to generate the final output, this could be a indirect learning from ground truth labels acoress the student models. Besides the efficiency advantages, I wonder whether it is more effective to directly use the parameter-efficient fine-tuning.\n* The curriculum learning has been widely studied in the literature of KD and I wonder whether it is necessary to include it in the meta-learning-based KD. The authors should provide detailed explaination on this with more ablation studies.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "b6qBwW56uj", "reviewer": "Reviewer_upPK", "summary": "Meta-learning-based knowledge distillation methods is a family of distillation techniques in which the teacher model is taking into account the student's performance to improve its learning. This paper introduces a new meta-learning-based knowledge distillation method which:\n(i) introduces a meta-teacher model with takes as input the original teacher's and student's representations and outputs improved predictions;\n(ii) introduces a reinforcement learning-based “student curriculum learning\" process in which the student aims to outperform the meta-teacher by training itself on a suitable set of tasks, possibly different from the task at hand.", "questions": "I don't have any questions.", "limitations": "", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "This is a well-written paper that introduces a novel technique that often times gives large improvements in the student's performance. The authors have made an extensive experimental evaluation comparing their results with other distillation techniques to give convincing arguments about their claims.", "weaknesses": "— If my understanding is correct, it seems to me that the main benefits come from the  “student curriculum learning\" process that exploits data points from other tasks that are in general not available to the other baselines the authors compare against and the teacher model itself. In that sense, it is not so surprising that the student model outperforms the teacher and that the improvements margins are high enough — this should probably be made a bit more explicit by the authors. (Indeed, it is known that the student-model can certainly outperform the teacher-model in scenarios it has access to more examples, see e.g., [1, 2, 3]). \n\n— The proposed technique is somewhat involved (implementation-wise).\n\n[1] Self-training with Noisy Student improves ImageNet classification [Xie, Luong,  Hovy, V. Le]\n[2] Does Knowledge Distillation Really Work? [Stanton, Izmailov, Kirichenko, Alemi, Wilson]\n[3] Weighted Distillation with Unlabeled Examples [Iliopoulos, Kontonis, Baykal, Menghani, Trinh, Vee]", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "sGZ090n7Wj", "reviewer": "Reviewer_cXqE", "summary": "This paper presents MPDistil, a meta-policy knowledge distillation framework for language models. The motivation is to tune both teacher and student models in a collaborative way. The proposed method consists of 4 steps, teacher fine-tuning, student distillation, meta-teacher learning, and student curriculum learning. The method is compared with the state-of-the-art baselines on SuperGLUE tasks and shows advantages. In several reported results, the teacher model outperforms the original teacher models, and the student model also outperforms the original teacher models. Finally, several discussions are presented with experiments.", "questions": "1. Have you tried using the meta-teacher as the teach in the baselines?", "limitations": "", "rating": 8, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "1. The paper is easy to follow, and the methodology is well-motivated.\n2. The method is compared with the state-of-the-art baselines, and the method shows advantages.\n3. Ablation study is provided to help understand the contribution of each design.\n4. Several interesting discussions are presented.", "weaknesses": "1. Given the previous work of Zhou et al. (2021), the presented method is not very novel, since the basic idea of teaching teachers is the same.\n2. The performance gain could be mostly attributed to a better teaching model. It would be better to conduct some cross-verifications by using the meta-teacher obtained in the proposed method as the teachers for other models.\n3. Larger language models are encouraged to be used in the experiments to better show the value of distillation.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (poster)