Towards Efficient Pre-Trained Language Model via Feature Correlation Distillation
Authors: Kun Huang
Xin Guo
Meng Wang
Venue: NeurIPS 2023 poster (2023)
Keywords: Knowledge Distillation; Pre-Trained Language Model
TL;DR: A simple yet effective feature correlation distillation for large Transformer-based PLMs.

Abstract: Knowledge Distillation (KD) has emerged as a promising approach for compressing large Pre-trained Language Models (PLMs). The performance of KD relies on how to effectively formulate and transfer the knowledge from the teacher model to the student model. Prior arts mainly focus on directly aligning output features from the transformer block, which may impose overly strict constraints on the student model's learning process and complicate the training process by introducing extra parameters and computational cost. Moreover, our analysis indicates that the different relations within self-attention, as adopted in other works, involves more computation complexities and can easily be constrained by the number of heads, potentially leading to suboptimal solutions.
  To address these issues, we propose a novel approach that builds relationships directly from output features. Specifically, we introduce token-level and sequence-level relations concurrently
  to fully exploit the knowledge from the teacher model. Furthermore, we propose a correlation-based distillation loss to alleviate the exact match properties inherent in traditional KL divergence or MSE loss functions. Our method, dubbed FCD, presents a simple yet effective method to compress various architectures (BERT, RoBERTa, and GPT) and model sizes (base-size and large-size).
  Extensive experimental results demonstrate that our distilled, smaller language models significantly surpass existing KD methods across various NLP tasks.

Towards Efficient Pre-Trained Language Model via Feature Correlation Distillation
Kun Huang'
Xin Guo' Ant Group
Meng Wang!*
(hunterkun.h hk, darren. wm, bangzhu. gantgrrouu...co
Abstract
Knowledge Distillation (KD) has emerged as promising approach for compressing large Pre-trained Language Models (PLMs). The performance of KD relies how to effectively formulate and transfer the knowledge from the teacher model the student model. Prior arts mainly focus directly aligning output features from the transformer block, which may impose overly strict constraints on the student model's 's learning process and complicate the training process by introducing extra parameters and computational cost. Moreover, our analysis indicates that the different relations within self-attention, as adopted other works, involves more computation complexities and can easily be constrained by the number of heads, potentially leading to suboptimal solutions. To address these issues, we propose a novel approach that builds relationships directly from output features. Specifically, we introduce loken-level and sequence-level relations concurrently to fully exploit the knowledge from the teacher model. Furthermore, we propose correlation- based distillation loss alleviate the exact match properties inherent in traditional KI. divergence or MSE loss functions. Our method, dubbed FCD, presents a simple yet effective method to compress various architectures (BERT, RoBERTa, and GPT) and model sizes (base-size and large-size). Extensive experimental results demonstrate that our distilled, smaller language models significantly surpass existing KD methods across various NLP tasks.
Introduction
Past few years have witnessed a rapid development of pre-traineds language models (PLMs) thanks to their effectiveness across a wide range of natural language processing tasks. Pre-trained language models, such as BERT Devlin et al. 2018). RoBERTa Liu et al. 2019). and GPT-2 Radford et al. (2019) learn contextualized text representations by predicting words given their context using large scale text corpora, and can be fine-tuned with additional task-specific layers to adapt to downstream tasks. However, the excellent capability for various NLP tasks comes at demaning huge resources and large memory footprints. For example, the BERT model contains about 110M parameters and 12 Transformer Vaswani et al. (2017) layers, which prevents these transformer-bes being finetuned and deployed on resource-constraine
models from devices and real-time applications. Recent studies Kovaleva et al. 2019), Voita et al. 2019 indicate that redundancy exists in the original PLMs. Therefore, series of attempts Chung al. 20200 Wu et al. 22000) Wang et al. (2020c). Gordon et al. 2020a Tang et al. 2019 Aguilar et al. [019 have been made to review the techniques for effective compression of the pre-trained heavy transformers without compromising the performance, of which knowledge distillation is considered to be a practical paradigm, Typically, knowledge distillation techniques aims at effectively transferring the dark knowledge embedded in large teacher network to boost the performance the smaller student network during training. Once trained, this compact
"Coreespodiing author.
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

student network can be directly deployed in real-life applications without introducing extra inference time or structure modifications. The essence of knowledge distillation relies on how to formulate and transfer the knowledge from teacher to student. The classic logit- based distillation Hinton et al. [0015)
directly mimic the final prediction outputs between the teacher and student via Kullback-Leibler (KI.) divergence, which only brings limited performance gain to the student. Besides this vanilla knowledge distillation, many other works Du et al. [2020). Heo et al. [2019). Tian et al. [2019) also try to make use of intermediate representations of the pre-trained teacher transformer. The intermediate layers contain more embedding, supplement richer features, thus allow the student transformer to acquire more information in addition to outputs. Diao et al., [2099] proposed TinyBERT, which distill the information between multiple intermediate features, including the embedding, self-attention matrices and hidden states of the teacher and student networks via the mean squared error (MSE). However, this usually need the adaption layers to align the mismatching embedding dimensions. Such gap makes it hard for the student mimic the teacher's feature directly and induces additional training cost as a consequence. MINILM Wang et al. [2000b] employs self-attention heatmaps and and value relations via the KL--divergence loss to deeply mimic teacher's self-attention modules. It leads to restriction that the number of attention heads of student model has to be the same as its teacher. To solve this problem, MINILMv2 Wang et al. [2020a] first concatenate and then split self-attention vectors of different attention heads according to the desired number of relation heads, which involves more computation of queries. keys, and values in self-attention and consequently leads to suboptimal performance.
Motivated by these observations, we aim to directly model feature relationships between the teacher and student models. In a manner similar to the self-attention mechanism, we first model token relations using the output features of the teacher and student models. This token-level relationship has the capacity to capture long-term dependencies between input tokens and highlight critical tokens essential for linguistic comprehension. While the token-level relation is intuitive, it reflect only one aspect of the feature relationships. We further exploit another important aspect of feature relations, the sample-level, to capture the semantic relationship across batch of samples, an aspect that has been overlooked in previous works. In this way, both types of relations are combined to complement each other to bring more fine-gras aeeeggaaied feature knowledge. Therefore, the student model is expected to have superior performance compared to that trained stand-alone. Thanks to the same shape of feature relations between student and teacher, our method offers increased flexibility with respect to the embedding dimension and the number of attention heads. Moreover, we propose a correlation- loss function to replace the KI. divergence and MSE used in traditional KD methods. Specifically, we employ the pearson linear correlation as a novel loss function, relaxing the exact match property typically associated with KL divergence and MSE. The overview of the proposed method is illustrated by [ To sum up, our main contributions are outlined as follows:
We directly model feature relationships between teacher and student models, which jointly leveraging loken-level and sample-leve relations to distill knowledge for the first time.
We propose a correlation loss function using Pearson linear correlation, and theo- retically explain that it offers more flexible alternative to traditional KL divergence and MSE.
Extensive experiments are conducted with popular variants of PLMs, including BERT, RoBERTa, and GPT on GLUE datasets, and our proposal consistently performs better than existing methods.
2 Related Works
2.1 Pretrained Language Models
Pretrained language models are pretrained on large amounts of text corpus, and then fine-tuned on task-specfic dataset. BERT Devlin et al. [2018) proposes to use a masked language modeling (MLM) objective to pretrain a deep bidirectional Transformer encoder. RoBERTa Liu et al. [2019] achieves strong performance by training longer steps using large batch size and more text data. Besides those encoder-based based models, GPT-2 Radford et al. [0119] is a decoder-based model designed for uni- directional, left-to-right text processing. It predicts the next word in a sequence given the preceding words, which allows to generate coherent and contextually relevant text. In additon to knowledge distillation, the compression of pretrained language models have been widely explored, ranging from

---

Student (8mall Model)
Teacher (Large Model)
Takeniowi
Predict Layer
Dot-Product
Dot-Prrdact
Prodict Layer
Balstion Transfy JPLGI
Add 5, Nor
Add & Norm
Food Forward
Piy
Pus
Feed Forward
Nor
Taken-lovel Rolation
Token-level Relation
Mull-Haed Salf Saf-Allattion
Sample
Sample lovel
Muti Self- Attertioon
Relation
Trunaler
X
Rat (220gt
X.
Layer
YPL
Embeddting Layer
Sample-avel (2h11)
Figure 1: Overview of the proposed method Feature Correlation Distillation (FCD). To demonstrate the effectiveness of FCD. We introduce the token-level relationship and sample-level relationship to distill the knowledge from teacher student. The correlation loss based pearson linear correlation is used to capture the relationship between the teacher and the student (best viewed in color).
unstructured pruning Gordon et al. 2020b), Guo et al. (201)), attetnion head pruning Michel et al.
[019];; to layer factorization Lan al. [2019], quantization Zhang al. [2020] Bai al.) 20200] and dynamic width/dpeth inference Hou et al. [020]. However, some of the techniques like weight
pruning (irregular sparsity) and quantization typically require complex piplines and can not lead to
inference speedup and run- time memory saving directly without dedicated bardware/librriie (e.g. for sparse or low-bit computing operation). By contrast, knowledge distillation has been found be a simple and much effective model compression technique that allows a relatively simple model to perform tasks almost accurately as complex model. Moreover, it can be combined with other compression techniques (i.e., where the student model is smaller, quantized, or pruned version of the teacher model) further compress the pre-trained language models.
2.2 Knowledge Distillation
Knowledge Distillation (KD) is process that transferring knowledge from a large teacher model to small student model. It was first proposed by Hinton et al. [2015) and then how effectively transfer more knowledge has been explored by many subsequent works Romero et al. 20044). Ahn
ct al. 2019]. Park et al. [0119] Tian et al. 2019). Tung and Mori 2019]. The intermediate layers contain much richer representation, thus allow the student transformer to acquire more information in addition to outputs. Tang et al. [209 distill fine-tuned BERT into extremely small bidirectional LSTM. Ture et al. [009]] initialize the student with a small pre-trained LM during taec-specific distillation. [Sun al. [2019a] introduce the hidden states from every k layers of the teacher to perform knowledge distillation layer-to-layer. [guulaar et al.) [2019) further introduce the knowledge of self-attention distributions and propose progressive and stacked distillation methods. Task-specific distillation requires to first fine-tune the large pre-trained LMs on downstream tasks and then perform knowledge transfer. The procedure fine-tuning large pre-trained LMs is costly and time-consuming. especially for large datasets. MiniBERT Tsai et al. [2019] uses the soft target distributions for masked language modeling predictions guide the training of the multilingual student model and shows its effectiveness on sequence labeling tasks. DistilIBERT [Sanh et al. (2019) uses the soft label and embedding outputs of the teacher train the student. TinyBER' Tiao et al. [2019] and MOBILE BEETERRTEE [Sun et al. (2019b) further introduce self-attention distributions and hidden states to train the student. For example, MOBILE-BERR
employs inverted bottleneck and bottleneck modules for teacher and student to make their hidden dimensions the same. TinyBERT uses uniform-strategy to map the layers of teacher and student when they have different number of layers, and a linear matrix is introduced to transform the student hidden states to have the same dimensions the teacher. However, the presence of those extra modules not only adds burden on network complexity but also complicates the training procedure.
3 Method
In this section, we first describe the transformer architectures and define the distillation target in Section 3.1 Then we introduce the two proposed types of feature relationships: token-level relation
3

---

Token
Tokon
Token
Sample
(Bena)
Token
(R
Bomple
Sample
Token-level Relation
Sample-level Relation
Our method proposes to maintain the token-level and sample-level relations between student and teacher. Token-level relation: relation between the tokens within each sample of teacher and student. Sample-level relation: relation between the samples on a specfic token.
Figure 2:
in Section 3.2 and sample-level relation in Section 3.2
At last, we elaborate on the correlation-bssdd loss Section 3.3 and provide theoretical analysis in Section
3.1 Preliminaries
The majority of pre-trained language models are based on Transformer architectures, which are composed of stack of Transformer blocks. We first tokenize the input sample into sequence of tokens and pack them together with special tokens such as [SEP] and [CLS]. These tokens are then projected into token embeddings and fed into transformer blocks. Each Transformer block consists Multi-Hes Head Attention (MHA) and a Feed- Feed-Forward Forward network (FFN). Layer normalization (LN) et al. (2016) and residual connection He et al. (016) are integrated around each of these two sub-blocks. Suppose a teacher model 7 and student model S, the model takes the feature Fi of the i-th Transformer block as input, In multi-head attention, heads are computed in parallel to get the final output, which can be formulated as:
A
Q1, Q1,K) Ki)
H1 = Σ A/Vi
H
Ar E RRHNNN is the attention matrix, where H denotes the number of heads and the sequence length of the input. It is calculated scaled dot-product between QI and K₁ and then apply softmax operation on the each column of matrix A,. The final multi-head attention output Hi is calculated as a weighted sum of values Vj. Suppose the two linear layers in FFN parameterized by W1, b and W2, 62, the output of FFN can be formulated as:
Fi = GeL + b,) + b2
We term F, E N as the output feature of the I-th Transformer block, where D denotes the dimension of hidden features. Some works directly adopted Fi as the distillation target. However, the embedding dimensions Ds and Dr of student and teacher are typically different. Previous works/Romen et al. (2014), Yim et al. [2017] Heo al. (0199 overcome this obstacle by building certain adaptation modules between hidden layers of the teacher and student models. However, these adaptation modules, with random initialization or special non-parameter transformation Srinivas and Fleuret 2018), Komodakis and Zagoruyko (011)] would potentially disturb training process, because it introduces extra parameters and computational cost (including weights, gradients and optimizer states) Pudipeddi et al. [2020] Moreover, the teacher and student models usually have different number of heads, i.c., Hr # Hs. To address these challenges. we aim to model feature relationships using Fi to overcome the aforementioned issues. The details of this process provided in the following sections.
Distillation with feature relationships
Token-level Relation To mitigate the negative influences of magnitude differences, we first normal-
ize the block features F of both the teacher and the student, denoted fF Norm(F). Common

---

normalization methods include é2, softmax and layer normalization. In our implementation, we choose lg normalization as its implementation. as consistently outperforms the other methods in our experiments. Subsequently, with the normalized feature, we compute the token-level relation matrix. Specifically, for a specific sample i, we define the relation matrix between tokens for each sample as follows:
F1.::
Fr.:: Tr (1FF....1
(x(F) = ||Fr:alle
(3)
Here, T' denotes transposition for the feature. For each given sample, both the student's and teacher's token-level relation matrices share the same dimensions of R, RNXN, Each matrix effectively serves as a relevance map, revealing the influence of each token in relation to others within the same sequence (see the left part of Figure 2). By doing so, this inner token-level relationship is adept at capturing long-term dependencies between tokens, emphasizing those which are crucial for comprehensive understanding of the linguistic context. Consider tasks such text tagging or named entity recognition, where each token must be assigned a label based on its role within the sentence. In such cases, the token-level relation matrix can serve an effective guide, assisting the student model to deliver enhanced performance.
Sample-leve Relation addition to the token-level relation, which captures the relationships among different tokens within each sample, the relationships across multiple samples for each token also provides informative knowledge for the distillation process. As such, we aim to distill this sample-level relation as well to enhance performance. Similarly, we compute the relation matrix at the sample level as follows:
F.je (F) 11F (
F:J:: (1F.s.l11)
Tr
In this case, for a specific token, both the student's and teacher's sample-level relation matrices, denoted as R,, share the same dimensions of IR BxB These matrices describe the relationships between different samples, as illustrated in the right part of Figure They enable the capture of semantic relationships across batch of samples, which is particularly essential for tasks like text classification and summarization. In these tasks, recognizing the sample-level relationship can be instrumental in making accurate predictions. It allows the model to discern the similarities and differences between samples, rather than viewing each sample isolation. For the sample-level relation modeling, instead of directly comparing words based on their positions across sentences, we transform each sentence into a unified high-dimensional space using the same network. Within this space, token features the same positions from distinct sequences become comparable, This approach forms the foundation of our sample-level relation modeling and serves as a valuable guide for the student model, enhancing its performance in tasks necessitating a deep understanding of inter-sample relationships.
Computation Complexity Analysis The token-level and sample-level relation maps, in pro- posed method, require computational complexities of 2B and 2B2 D., respectively. The asso- ciated memory space required for these computations is BN B²N. In contrast, MiniLM Wang et al.] (2020b) utilizes self-attention and value-value attention, results in total computational complexi- ties and memory space requirements of 4BN2 and 2BN2 respectively. MiniLMv2 Wang et al. (2020a) employs a different attention mechanism, resulting in total computational complexities and memory space requirements of 1 D and 9B.22 respectively. For concrete example, consider the BERTBASE model, where B is set to 32, N is 128, and D is 768. Our proposed method yields computational complexities and memory space requirements of 0.6GFLOPs and 0.6MB, respec- tively. In comparison, MiniLM has computational complexities and memory space requirements of 1...GFF.PPP and 2MB, respectively, while for MiniL.My2 these values rise 14.5GFL.OP and 9.4MB, respectively. Thus, our method demonstrates significant efficiency in terms of computational complexity and memory space requirement, providing a more resource-efficien option for knowledge distillation.
3.3 Distillation with pearson linear correlation
In addition the distillation target, the design of the distillation loss plays crucial role transferring the knowledge from the teacher model to the student model. A general distillation loss can be
5

---

expressed as:
C C=r(rr(Fr) os((s)
(5)
Here, Φr and Øs denote the feature transformations for the teacher and student models, respectively. These transformations convert raw features into a form that is more conducive to knowledge transfer. In previous works, often takes the form of an adaptation layer that aligns the embedding dimensions between the teacher and student models. However, in our method, as discussed in Section 3.2 is used denote the token-level and sample-level relation matrix. / is distance function measuring the similarity between student and teacher features. The most commonly used functions include KL divergence and the mean squared error. However, these functions exhibit "exact match' property, which means the loss reaches the minimal if and only if the features of student and teacher are all identical. This requirement could impose overly strict constraints on the student, particularly when there is a large discrepancy between the teacher and student. To alleviate these constraints, we introduce the Pearson correlation coefficient as a more flexible alternative, which is used measure the strength of the linear relationship between two variables and remains invariant under positive linear transformations. The basic form Pearson correlation coefficient between X and follows the Pearson index can be computed as follows:
p(X,)) = -(X:-MXX1 - -YY)
- -(y--My
where μ.. . jy denote the mean value of the variable X, Y respectively. The Pearson correlation coefficient, denoted by p. ranges between and 1. Therefore, making it suitable to be used as loss function. We call our final formulation of the distance function Pearson linear correlation (PLC), which is defined as follows:
P always falls between 0 and
T(X, Y) = 1===-1pp))
In this way, we shift the focus from attempting to exactly replicate the teacher's features to preserving and learning the relational information between the teacher's and student's features. This shift effectively relaxes the 'exact match' requirement inherent in conventional Knowledge Distillation (KD) methods. Thus, the distillation loss in our approach comprises two types of losses: token-level relation loss and sample-level relation loss. The token-level relation loss quantifies the discrepancy between the token-level relation matrices of the student and teacher models. It is defined as the average PLC between the student's 's and teacher's token-level relations across all samples in the batch:
1 B B
L. G:= I
8
The sample-level relation loss quantifies the discrepancy between the sample-level relation matrices of the student and teacher models. It is defined as the average PLC between the student's and teacher's sample-level relations across all tokens in the sequence.
N := = N E (RR.
The intrinsic sensitivity of the PLC to outliers necessitates normalization of features prior to calculat- ing the distillation loss, thus ensuring stable training process. Given these two components, the overall training loss of our proposed method consists of the task loss, token-level relation loss, and sample-level relation loss, which can be formulated as follows:
-= Lg + aLt + 32.
(10)
Here, L, denotes the task training loss, while a, 3 are weighting factors used to balance the task training loss and the relation losses.
3.4 Theoretical Analysis
As discussed in Section 3.3 different distance functions like Kullback-Leible (KL) divergence and Mean Squared Error (MSE) are commonly used to measure the similarity between the teacher and the student models. However, these functions have an 'exact match' property, which means that the distance is zero if and only if the student and teacher features exactly the same. This

---

Model
#Params
Speedup
SST-2
MNLI-m
QNLI
QQP
RTE
SST-B
MRPC
CoLA
AVG
BERTRASE
110M
x1.0
93.4
84.5
91.5
72.3
66.8
85.2
88.3
52.8
BERT SMALL
66M
x2.0
90.7
81.2
87.9
69.4
64.3
79.8
83.7
41.4
74.8
BERT- &
66M
x2.0
92.0
81.5
89.0
70.7
65.5
81.6
85.0
43.5
76.1
DistiBBRTTa
66M
x2.0
92.5
82.6
70.1
58.4
86.9
49.0
TinyBERT,
66M
x2.0
92.1
82.8
89.8
71.2
70.0
83.9
88.0
51.1
78.6
MINILMG
66M
x2.0
92.0
83.0
91.1
71.4
70.8
84,2
88.5
49.2
78.8
Mv,
92.4
83,4
71.5
71.3
51.8
Ours
66M
x2.0
92.8
83.8
91.3
72.0
71.7
84.8
89.1
52.0
79.6
RoBERTaBAsE
125M
x1.0
95.3
87.2
93.2
73.8
72.7
88.4
90.1
62.0
82.8
82M
x2.0
92.3
83.1
90.4
72.1
68.4
86.8
87.5
54.1
79.3
MVN
82M
x2.0
93.5
84.3
72.8
72.1
87.5
88.2
57.8
81.0
Ours
82M
x2.0
93.8
85.6
92.0
73.5
72,5
88.3
89.6
60.3
81.9
DistiGPT2
82M
X2.3
90.7
81.6
87.9
66.8
68.3
79.6
87.9
39.4
75.3
x2.33
Table 1: Results of the proposed method on the test sets of GLUE. We use the metric of Matthews correlation for CoLA, Pearson- Sprermnnn correlation for STS-B, and accuracy for other datasets. Following previous works (un et al. [2019a]. we also report the average score of these eight tasks (the "AVG" column). The speedup is in terms of the BERT BARE BASE and RoBERTaBASE inference time and evaluated single GPU with single input of 64 128 length. The fine-tuning results average runs.
can be written as I) = when X = Y. Unlike KL divergence or MSE, PLC is invariant under positive linear transformations. This means that even if positive linear transformation is applied to one or both of the features, the correlation coefficient remains the same. In other words, T(X, Y) Y aX 3, where a > 0 and are constants. This property makes PLC more flexible choice for knowledge distillation, model less restrictive way. The detailed mathematical justification for this property is provided in the supplementary materials.
it allows the student model to learn from the teacher
We next delve into the relationship between the Pearson linear correlation (PLC), Kullback-Leibler (KL) divergence and Mean Squared Error (MSE). For KL divergence, the normalized features are transformed into probability distribution, and then we minimize the discrepancy between the softened probabilities of the teacher and student models.
(F(IT)
CRI = (F log (Fs:T)
(11)
Here, т denotes temperature parameter used to adjust the softness degree in the distributions. Assuming the value of significantly large compared the magnitude of the normalized features, and f are drawn from a standard normal distribution, we can derive the gradient of LK with respect to the normalized feature F as follows:
0LKL 1 0f
(fs-
--) 0fs
OLMSE
(12)
M
Further, given that - F3 and follows:
f3 = 1, we can reformulate the MSE loss as m-I E;
(F 2M
-
(13)
This equivalence clarifies the intrinsic connection between KL divergence. MSE, and PLC of nor- malized feature distributions. The detailed derivations and empirical results to support this claim are provided in the supplementary material.
Experiments
4.1 Distillation Setup
evaluate the efficacy of the proposed FCD on out of tasks from the General Language Understanding Evaluation (GLUE) Wang et al. 2219 benchmark, which consists of 2 single-sample

---

PCD
500000
MPPP
6518
ONL
GNI
WRPP
661-2
(4) BERT An the beacher
(6) RoBERTa as this Inacher
(c) Ablotion of takeniovel and semple level relations
Figure 3: Comparison of distillation results of student models using BASE-size and LARGE-size teachers (BERT and RoBERTa) shown in the left part. The right part shows ablation of token-level and sample-level relations.
(CoLA and SST-2) and 5 sample-pair (MRPC, QQP, MNLI, QNLI and RTE) classification tasks, and 1 regression task (STS-B). Following previous worksSun et al. 2019a], we use the same metrics as the GLUE benchmark for evaluation. In order to verify the effectiveness and robustness of our method, distill teacher models with different architectures, model sizes. Concretely, we consider encode-based models including BERT BERTLARGE. RoBERTaBASE and RoBERTaLARd to ensure a fair comparison with of a wide range of prior works. Moreover, we also explore the compression of decoder-based models by employing our distillation method to improve fine-tuning of DistilGPT2 Woll et al. [2020] which rarely investigated in most previous works.
4.2
Implementation Details
The process of distilling pretrained Language Models (LM) generally comprises two stages: task- agnostic distillation and task-specific distillation. Task-agnostic distillation involves a pre-training process large-scale dataset. However, this stage be costly and time-consuming particularly for scenarios with limited computational resources. In contrast, task-specfic LM distillation proves to be effective and considerably more economical compared to pre-training. As increasing number of pretrained LM models are becoming publicly available through resources such as the HuggingFace Transformers library Wolf al. [2020), directly leveraging these models can save substantial time and computational resources compared to training from scratch. Given these considerations, our focus in this work is on task-specific knowledge distillation. However, is important note that our method is not limited to task-specific distillation and can be readily applied in task-agnostic scenarios as well. Specifically, we first fine-tune the pretrained teacher models on specific task. Following this, the corresponding student model is initialized with the teacher model using the LayerDrop method (Sajjad al. [2020]. Subsequently, we perform distillation with FCD. We employ grid search algorithm on the development set to tune the hyper- parameters. Specifically, we trained the student model for 3, 5 and 10 epochs, using batch size of 32. The learning rates we experimented included 2e 5. le 5 and 5e 6. For the CoLA task, we extended the training steps to 25 epochs. The parameters a and 3 from the distillation loss are tuned from (0.1, 0.2, 0.4, 13, a choice guided by maintaining the different components of the loss in the same order of magnitude. We adopt cosine decay schedule with a warm-up phase of epochs and utilize the Adam W optimizer with a weight decay of 0.5. The maximum sequence length is set to 64 for single-sample tasks, and 128 for sequence pair tasks.
4.3 Main Results
We start by comparing our proposed method with several KD baselines, including DistilBERT Sanh ct al. [2019], TinyBERT Jiao et al. [2019]. BERT-PKD Sun et al. [2019a]. MiniLM Wang et al. [2020b], MiniLMv2 Wang et al. 2020a]. Similar to previous studies, we distill a 12-layer base model into a 6-layer student model with only about 60% parameters and 2x inference speedup. In order evaluate the impact of knowledge distillation, we also report the results of BERT BERT and RoBERTasMALLE- These smaller models are obtained using the Laypppgg method Sajjad al. 2220 wherein the strategy of dropping the top layer has been demonstrated to strong baseline. Consequently, we drop the top 6 layers of the base model and fine-tune it without using knowledge distillation. For fair comparisons, we fine-tune the released models and evaluate the result
on the test set of GLUE without resorting to data augmentation strategy Jiao al. [2019] The results

---

Table 2: Comparison of using different distillation loss functions.
Table 3: Distillation results with different layer selecting schemes.
Method
MNLI-m
QQP
RTE
Average
Scheme
layers
MNLI-m CoLA
FCD (KL)
83.3
71.2
70.7
75.1
Top
17.8.9,10.11.12]
82.9
50.7
FCD (MSE)
83.6
71.6
71.2
75.5
Uniform 12.4.6,8.10.12]
83.8
52.0
FCD (Pearson)
83.8
72.0
71.7
75.8
Bottom
(1,2,3.4.5.6)
83.3
51.5
from these 6-layer student models are summarized in Table [ The top group of models denotes the uncased version of basc-size BERT used as the teacher model.Notably, compared models by a large margin. Similar trends are observed the middle group, where base-size RoBERTa is utilized the teacher model. Our model surpasses the MiniLMv2 by 0.4% accuracy on RTE, 0.5% F1 on MRPC, and 0.3% Spearman correlation on STS-B. Furthermore, we conduct experiments
model outperforms all
decode-base model DistilGPT2. In this setting, employ the GPT-2 model with 12 layers and 768 hidden size as the teacher model. Noatably, our proposed method outperforms the original KD method by an average of 1 1%,, underlining the efficacy of our approach in decoder- based model. As illustrated in the left part of Figure 3 student models that are distilled from large-size teacher models achieve further improvements. Moreover, this performance gain increases with the capacity of teacher models, thereby demonstrating the effectiivneess of our proposed method across different sizes of pretrained Transformer models.
4.4 Ablation Studies
Effect of different components In this study, we introduce two distinct types of relations: token- level and sample-level relations. To verify the effectiveness of each, we conduct experiments using La and L. to investigate their respective influences on the student model. As depicted the right part of Figure 3 each component within the distillation loss independently contributes to the enhancement of the final performance. Moreover, a further boost in performance is observed when these components are combined. We noticed that the performance degradation on SST-2 is more substantial compared to other tasks without token-level relation. We speculate that the token-level relation particularly important for this single sentence binary classification task.
Effect of different distillation loss functions Here, we compare proposed Pearson Linear Correlation (PLC) with the Mean Squared Error (MSE) and Kullback-Leibe (KL) divergence, which are widely-used loss functions. To ensure a fair comparison, tune the distillation loss weight for both MSE and KI.. The comparative results across three tasks are presented in Table [2] which demonstrates that adopting Feature Correlation Distillation (FCD) with PLC consistently yields higher performance compared to the FCD combined with MSE and KL. This indicates that the more flexible Pearson correlation might serve more suitable metric for measuring relations within the FCD.
Effect of different layer selection strategies Apart from the type of knowledge used for distillation, the selection of layers significantly affects the overall performance of distillation. We study the impact of three distinct layer selection strategies: uniform, top, and bottom, and compare their respective performances. Specifically, we utilize BERTBAST BASE as the teacher model and a 768 model as the student model. The number of selected layers is set to 6. The results are reported in Table 3 For BERT BASE, using the uniform layer selection strategy yields superior performance compared the other strategies. This finding highlights the crucial role of both the head and tail layers of student models in the distillation process.
5 Conclusion
In this paper, we introduce Feature Correlation Distillation (FCD), novel and effective method for distilling large Transformer-baser Pretrained Language Models (PLMs). Our approach simultane- ously models both token-level and sample-level relations derived from the features of the Transformer block. Moreover, we propose correlation- based distillation loss to enhance the performance of the model distillation process. We also provide theoretical interpretation of our proposed Pearson

---

linear correlation formulation, offering deeper understanding of its underlying operation and impli- cations. Through extensive experiments on the GLUE tasks, our distilled smaller language models consistently outperformed existing knowledge distillation methods across a variety of architectures while significantly reducing both the model size and inference time. With its simplicity and strong performance, we hope our approach can serve as solid baseline for future research.
Acknowledgements
This research was supported by Ant Group. We are grateful to the five anonymous reviewers for their insightful suggestions and comments, which significantly enhanced the quality of this paper.
References
Gustavo Aguilar, Yuan Ling, Yu Zhang, Benjamin Yao, Xing Fan, and Edward Guo. Knowledge distillation from internal representations. CoRR, abs/1910.03723, 2019. http://///rrxx http://wxxx...rgabbbb///b wttp:oo//////../e. 13900
Sungsoo Ahn, Shell Xu Hu, Andreas Damianou, Neil D Lawrence, and Zhenwen Dai. Variational information distillation for knowledge transfer. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 9163-9161191111 9171,9911,22111 2019.
Lei Jimmy Ba, Jamie Ryan Kiros, and Geoffrey E. Hinton. Layer normalization. CoRR, abs/1607.064500 2016.
URL http://arxiv org/aaagga abs/ 1607. .64450
Haoli Bai. Wei Zhang, Lu Hou, Lifeng Shang. Jing Jin, Xin Jiang, Qun Liu, Michael Lyu, and Irwin King. Binarybert: Pushing the limit of bert quantization. arXiv preprint arXiv:2012.1570.1 2020.
Insoo Chung. Bycongwook Kim, Yoonjung Choi, Se Jung Kwon, Yongkweon Jeon, Baeseong Park. Sangha Kim, and Dongsoo Lee. Extremely low bit transformer quantization for on-device neural machine translation. In Findings of the Association for Computational Linguistics: EMNLP 2020, pages 4812 1822,486,, Online, November 2020. Association for Computational Linguistics. doi: 10.1
1en220iinningg-emm---5535 emnlp.433. httph/tta////ww... aww.wb..l.......... org/ attho/nntooooggg ogy/ 2020 findings- emnlp 111..p..... .433
Jacob Devlin, Ming Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre- tre-training of deep bidirectional transformers for language understanding, arXiv preprint arXiv:1810.04805. 2018.
Shangehen Du, Shan You, Xiaojic Li, Jianlong Wu, Fei Wang, Chen Qian, and Changshui Zhang. Agree to disagree: Adaptive ensemble knowledge distillation in gradient space. advances in neural information processing systems, 33: 12:12345- 12355,, 2020.
Mitchell Gordon, Kevin Duh, and Nicholas Andrews. Compressing BERT: Studying the effects of weight pruming on transfer learning. In Proceedings of the 5th Workshop on Representation Learning for NLP, pages 143-155, Online, July 2020a. Association for Computational Linguistics. doi: 118866/1112200.rp/gp 1.118. URL htttps://ww/ wwwp:///www.wwwwbb......
aclweb. .rg. httg://nthooogyy// 2020 rep14n1p-1.111
Mitchell Gordon, Kevin Duh, and Nicholas Andrews. Compressing bert: Studying the effects of weight pruning on transfer learning, arXiv preprint arXiv:2002.08880 2020b.
Fu-Ming Guo, Sijia Liu, Finlay Mungall, Xue Lin, Yanzhi Wang. Reweighted proxim al pruning for large-scale language representation. arXiv preprint arXiv: 12486, 2019.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In 2016 IEEE Conference on Computer Vision and Pattern Recognition. CVPR 2016, Las Vegas, NV, USA, June 27-30, 2016, pages 770-778, 2016.
Bycongho Hco, Jeesoo Kim, Sangdoo Yun, Hyojin Park, Nojun Kwak, and Jin Young Choi. A comprehensive overhaul of feature distillation. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 1921-1930 2019.
Geoffrey Hinton, Oriol Vinyals, Jeff Dean, et al. Distilling the knowledge in a neural network. arXiv preprint 1503.00503 2(7), 2015.
Lu Hou, Zhiqi Huang, Lifeng Shang. Xin Jiang, Xiao Chen, and Qun Liu. Dynabert: Dynamic bert with adaptive width and depth. Advances in Neural Information Processing Systems, 33:9782-9793, 2020.
Xiaoqi Jiao, Yichun Yin, Lifeng Shang. Xin Jiang. Xiao Chen, Linlin Li. Fang Wang, and Qun Liu. Tinybert: Distilling BERT for natural language understanding. CoRR, 103551, 2019. URL http://arxiv org/abs/ 190ee//999....0 10351
10

---

Nikos Komodakis and Sergey Zagoruyko. Paying more attention to attention: improving the performance of convolutional neural networks via attention transfer. In ICLR, 2017.
Olga Kovaleva, Alexey Romanov, Anna Rogers, and Anna Rumshisky. Revealing the dark secrets of BERT.
In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th
International Joint Conference on Natural Language Processing JEMNLP-IJCNLL pages 4365-4374, Hong
Kong. China, November 2019, Association for Computational Linguistics. doi: 10.18653/11/D19-14455 URL
httpa://www www.:e..wwlbb..o. org/ onthology/D19-1433 httpp///www..llllleo.orrrraaaaaa
Zhenzhong Lan, Mingda Chen, Schastian Goodman, Kevin Gimpel, Piyush Sharma, and Radu Soricut. ALBERT: A lite bert for self-supervised learning of language representations. ArXiv, abs/1909. 1942, 2019.
Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. RoBERTa: A robustly optimized BERT pretraining approach. arXiv preprint arXiv: 1692, 2019.
Paul Michel, Omer Levy, and Graham Neuhig. Are sixteen heads really better than one? Advances in neural information processing systems. 32, 2019.
Wonpyo Park. Dongju Kim, Yan Lu, and Minsu Cho. Relational knowledge distillation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pages 3967-3976, 2019.
Bharadwaj Pudipeddi, Maral Mesmakhosroshahi, Jinwen Xi, and Sujeeth Bharadwaj. Training large neural networks with constant memory using new execution algorithm. arXiv preprint arXiv:2002.05644 2020.
Alec Radford, Jeff Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. Language models unsupervised multitask learners. 2019.
Pranav Rajpurkar, Jian Zhang. Konstantin Lopyrev, and Percy Liang. SQuAD: 100,000+ questions for machine comprehension of text. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pages 2383-2392. Austin, Texas, November 2016. Association for Computational Linguistics. doi: 10. 1/D1 1264. URL https://t// acl wew.eb.wbbbb....cc......... org/ anthology/DD1 1264
Pranav Rajpurkar, Robin Jia, and Percy Liang.
Know what you don't know: Unanswerable questions for SQuAD. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 784-789, Melbourne, Australia, July 2018. Association for Computational Linguisties. doi: 10. 10.18853/v1/P118 2124. URL/atpss://ww/ aclweb. org/ /tt/////tooollo/////222244
Adriana Romero, Nicolas Ballas, Samira Ebrahimi Kahou, Antoine Chassang, Carlo Gatta, and Yoshua Bengio. Fitnets: Hints for thin deep nets. arXiv preprint arXiv:1412.66500 2014.
Hassan Sajjad, Fahim Dalvi, Nadir Durrani, and Preslav Nakov. Poor man's BERT: smaller and faster transformer models. CoRR, abs/2004.0332 00844, 2020.
Victor Sanh, Lysandre Debut, Julien Chaumond, and Thomas Wolf. Distilbert, a distilled version of BERT: smaller, faster, cheaper and lighter. CoRR, abs/1910.01108, 2019. URL http:ttt///a/ri.orrr org/ ab911900 01108
Suraj Srinivas and François Fleuret. Knowledge transfer with jacobian matching. In International Conference on Machine Learning, pages 4723-4731. PMLR, 2018.
Siqi Sun, Yu Cheng, Zhe Gan, and Jingjing Liu. Patient knowledge distillation for BERT model compression. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing, EMNLP-IJCNLPP 2019, Hong Kong, China, November 3-7, 2019, pages 4322-4331, 2019a.
Zhiqing Sun, Hongkun Yu, Xiaodan Song, Renjie Liu, Viming Yang, and Denny Zhou. Mohilchert: Task- agnostic compression of bert by progressive knowledge transfer, 2019b. URL /RRLatttpe//pppnneiiee.ee SxxfiddSJxjjaNike.
Raphael Tang, Yao Lu, Linqing Liu, Lili Mou, Olga Vechtomova, and Jimmy Lin. Distilling task-specific knowledge from BERT into simple neural networks. CoRR, abs/1903.11136, 2019. URL http://arxii org/abs/ 1903. 12136
Yonglong Tian. Dilip Krishnan, and Phillip Isola. Contrastive representation distillation. arXiv preprint 1r01v:1900 2019.

---

Henry Tsai, Jason Riesa, Melvin Johnson, Naveen Arivazhagan, Xin Li, and Amelia Archer. Small and practical BERT models for sequence labeling. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing, EMNNP-IJCNE 2019, Hong Kong, China, November 3-7, 2019, pages 3630-3634, 2019.
Frederick Tung and Greg Mori. Similarity-reesevinng knowledge distillation. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 1365-1374, 2019.
Iulia Ture, Ming-Woi Mei Chang, Kenton Lee, and Kristina Toutanova. Well-read students learn better: The impact of student initialization on knowledge distillation. CoRR, abs/1908 05s/1908.08960 08962, 2019, URL http
//arxiv. owgxivoorg/aoe aba/ .89622
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones. Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Advances in Neural Information Processing Systems 30, pages 5998-6000 Curran Associates, Inc., 2017. URL http::/paperrs.eeepe// wwtp:/errr..pp...c.eeeeeeeeeeeee cc/pprr 1s-allllllllllllll all aol-you----- pouu-eed.ppl pdf
7181-
Elena Voita, David Talbot, Fedor Moiseev, Rico Semnrich. and Ivan Titov. Analyzing multi-head self-atiention: Specialized heads do the heavy lifting, the rest can be pruned. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 3797-5805 Florence, Italy, July 2019, Association for Computational Linguistics. doi: 10. 10.18653/v1/P19. 1580. httpss/s:://www http::::///www...llee... org/ //aaahhhooogyy
P19-1580
Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R. Bowman. GLUE: A multi-task benchmark and analysis platform for natural language understanding. In International Conference on Learning Representations, 2019. URL.tttee htttpp:://ppeneeiie wttpeeewwwwwweeeeeeeeeeeeeeeeeee www.:eetffooooe net/forummid=: J4km2R5t7
Wenhui Wang, Hangbo Bao, Shaohan Huang, Li Dong, and Furu Wei. Minilmv2: Multi-head self-attention relation distillation for compressing pretrained transformers. arXiv preprint arXiv: 2020a.
Wenhui Wang. Furu Wei, Li Dong, Hangbo Bao, Nan Yang. and Ming Zhou. Minilm: Deep self-attention distillation for task-agnoss agnostic compression pre-trained transformers. In Advances Neural Information Processing Systems 33: Annual Conference on Neural Information Processing Systems 2020, NeurIPS 2020,
December 12, 2020, virtual, 2020b.
Ziheng Wang, Jeremy Wohlwend, and Tao Lei. Structured pruning of large language models. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 6151-6162, Online, November 2020c. Association for Computational Linguistics. doi: 000653//120200.mmmmmmmo/ URL httLLhtta:////ee aclweb. org/ httgg//ntologyy/// 2020. emnlp-main 496
Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac. Tim Rault, Remi Louf, Morgan Funtowiez, Joe Davison, Sam Shleifer, Patrick von Platen. Clara Ma. Yacine Jernite, Julien Plu. Canwen Xu, Teven Le Scao. Sylvain Gugger, Mariama Drame, Quentin Lhoest, and Alexander Rush. Transformers: State-of ofate-of-the-art natural language processing. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, pages 38-45, Online, October 2020. Association for Computational Linguisties. doi: 1116655/1/22220.mmmm-ddaaaaa URL htttp:://wwtt acl web. org/ wwwtaholggyyaaana eem.:00essmm--ddmeee6
Xiaobao Wu, Chumping Li, Yan Zhu, and Yishu Miao. Short text topic modeling with topic distribution guantization and negative sampling decoder. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 1772-1782, Online, November 2020. Association for Computational Linguistics. doi:
1/2020.nn/n main. 138. URL httph:tt////// www. aclweb. org/ aanhhology22200 emnle-maneee.. waalppmmiin....m 138
Junho Yim, Donggyu Joo, Jihoon Bae, and Junmo Kim. A gift from knowledge distillation: Fast optimization,
network minimization and transfer lcarning. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 4133-4144 2017.
Wei Zhang, Lu Hou, Yichun Yin, Lifeng Shang, Xiao Chen, Xin Jiang, and Qun Liu, Temaryyeer: Distillation- aware ultra-low bit bert, arXiv preprint arXiv:2009. 128/2, 2020.
6 Appendix
6.1
Implementation of FCD
The implementation details of FCD are presented in Figure Utilizing the output features from both the student and teacher models, denoted as F, and F, respectively, our method ensures ease of implementation.
12

---

irport
from torch.ne inport functional
cos cosine_s sesiie_sinilarityyk y, eps=le-8);
y)-sum(1) (v..oo((die=1) y-nora(die=1) eps]
def prrson_correlatiinna, y, eps-le-8);
return cos wewiiiissilaattCCC. y-nean(-1) unsqueezzel-
def [oc_loss(,, y): return peerson_correlationn ion y)-iea)())
def taken, retattion loss(y_f. y_t):
R_S y.5.brmty...ttrassppcc. -213
R_T y_t.bbmdyy_--rranppaacc -2))
R_t R_t.vinw(R_t..s22000 -1)
toss ptc, toss(Oss, Rgt)
return loss
def sample relation tossly y.t):
Y_5 Y_s.pormutell, 0, 2)
y_t y_t-permute(1, 9,
loss relation lossty y.t) return loss
class FCOLoss Moduutttl):
def init init (self, alpha, beta): super(FCDloss, setf)._
self.alpha alpha self.beta beta
def tonerd(selt, F.s, F_t):
F.nomatize(FFF. dim--)
Funoratsze(FF.. din-1)
toss_token rooken_eelation_tte Loss((__, F_t)
toss_sonole sanpte somple_relation, loss(Fss, F_t)
ko_less self.alono toss_token
Loss_sample
return kd_loss
Figure 4: The PyTorch implementation of FCD.
6.2 Proofs
Invariance of Pearson's Correlation under Positive Linear Transformation. Let's consider two random variables X and Y. positive linear transformation on X and Y can be formulated as X' aX + b and Y' = cY + d, where a x с >> 0 and b, d are arbitrary constants. Applying these transformations to the means of and Y, we derive y = apx + and Jyy = cpy + By substituting the transformed variables and their corresponding means into Equation 6
E(lax, - 6) (aux + b))((( d) (cHy + d))
d(X,)) VI((X,, + (apx + b))3 ())) +
(epy - d))2
(14)
E(a(X, (x)) Hy))
x(X,Y)
Relationship among PLC, KL divergence and MSE In Equation 11 Φ(-) represents the softmax function, while denotes a temperature parameter controlling the softness of the distributions.
exp(Fb/7) 0 LKI. = - (0)(t)) = Σ exp(FP/TT af
exp(Ft/T) E (xp(Ft//T)
(15)
Assuming is significantly large compared to the magnitude of the normalized features and both F and F7 are drawn from a standard normal distribution. In this case, the term Fi/r /r becomes quite small, allowing approximate (xp(F/T) 1 //. This simplification leads to an
13

---

(b)
(q)
(d)
Figure 5: Visualization of the token-level relational features under different normalization functions. (a) Pre-Norm. (b) 22-Norm. (c) Layer-Norm. (d) Softmax-Norm.
approximation the gradient in Equation 15
0CKL
1+F5/T
1 1+ F//T
RT
(16)
0f3s
M -+
M
M+E, F4//T
Given that the sums Σ, F/ and E, F/. are both zero, Equation 16 simplifies further to:
0LKL.
1
(Fs - Fr) F
0LMSE
(17)
0j$ M
03%
Moreover, considering - E. and
- m-1 E. 1, we can reformulate the MSE as
follows:
LMSE (Fs
E - -F1)2
2M
1
(2M - 2) -2 > FsF
(18)
2M
2M - 2
2M (1 - ((F,, 2 (PLo(Fs,Fr)
Thus, we demonstrate that minimizing KL divergence between normalized features under high- temperature limit is equivalent minimizing the MSE between normalized ones, which is in turn equivalent to maximizing the PL.C between the original features.
6.3 More Experiments Results
Effect different Normalization and Loss Functions Section 3.4 in the main text clarifies the intrinsic relationship between KL divergence, MSE, and PLC. However, the assumption that normalized features follow a Gaussian distribution may not invariably be valid. To investigate the performance of varying normalization and loss functions, setting the temperature to 10 when utilizing KL divergence as the loss function. demontsdemsssssttes that the f2 norm consistently outperforms other normalization functions. Minimizing KL divergence between noryea-ordallzed features in the high-temperature limit can yield results comparable to MSE and PLC. To further underscore the benefits of /2 normalization, pre-normalized and post-normalized token-level relational features in Figure [5] In contrast to /2 normalization, other functions often produce a wider range with larger values, suggesting that directly imitating these normalized features could introduce significant noise, potentially leading to subpar results.
conducted series of experiments,
provide visualization of the
Results on SQuAD v1.1 and v2.0. To further demonstrate FCD's 's effectiveness, we applied it to the question-answent tasks of SQuAD v1.1 Rajpurkar al.) (2016) and SQuAD v2.0 Rajpurkar et al.) (2188. We framed these tasks as sequence labeling problems, predicting the likelihood of each token being the start or end of an answer span. We employed the Fl metric for both versions SQuAD. BERTRASE was used as the teacher model, and a 6 x 768 model served as the student model. The results, presented in Table indicate that FCD can enhance the student model's performance on both tasks.
14

---

Table 4: Results of the loss function combined with different normalization mechanism.
Scheme
MNLI-m
QQP
CoLA
Average
LayerNorm KL.
83.4
71.5
51.4
68.8
LayerNorm MSE
83.6
71.8
51.5
69.0
PreNorm + PLC
83.3
71.2
51.7
68.7
Softmax PLC l + PLC
83.4
71.7
51.5
68.9
83.8
72.0
52.0
69.3
Table 5: Results of baselines and FCD on question answering tasks.
Method
SQuAD 1.1
SQuAD 2.0
BERT BASE
88.7
78.8
DistilBERT,
86.2
69.5
TinyBERT,
87.5
77.7
Ours
88.2
78.4
6.4 Discussion
Limitations. While FCD demonstrates consistent performance improvements across diverse Transformer-base based models. its effectiveness may be less pronounced on other architectures such as Recurrent Neural Networks (RNNs). The feature relationships in RNNs are not as explicit as in Transformers, potentially limiting the applicability and impact of FCD.
Societal impacts. The extensive computational resources required to evaluate our proposed method could significantly contribute to carbon emissions, thereby raising sustainability concerns. However, the objective of our approach is to enhance the efficiency of lightweight models through knowledge distillation. This enhancement could ultimately replace heavier models in production settings, resulting in substantial energy savings. Thus, the thorough validation of FCD's efficacy is necessary trade-off to ensure its potential benefits.
15


Reviews:
[{"review_id": "jmrvZaWOE1", "reviewer": "Reviewer_qBd9", "summary": "This paper explores the task-specific knowledge distillation from a large teacher model which are pre-trained language models (PLMs) such as BERT-large or BERT-base into a student which is always smaller than the teacher model. For example, the teacher can be a 12-layer of BERT-base model while the student model only has 6-layer. One of challenges for knowledge distillation is that the feature dimensions and number of attention heads are different, and it requires additional transformation to match both the outputs of teacher and student models. \n\nAuthors propose to build token-level and sample-level relationship from feature representations, and then compare these relations between teacher and student models. They further advocated to use a correlation-based loss function over KL or MSE. The proposed approach is evaluated on the GLUE benchmark showing the effectiveness of their method. ", "questions": "- line 292 says it is trained for 20 epochs which seems much longer than standard finetuning. Is it because the proposed approach is slow to converge?\n\n- In Figure 4 (6.1), it is required to define B first before use in the function (token_relation_loss).\n", "limitations": "n/a", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "\n-\tThe idea of token-level and sample-level feature mapping between teachers and students sounds interesting.\n-\tIt also provides a strong empirical result. \n-\tAblation study also shows that importance of different components.\n", "weaknesses": "-\tIt is hard to understand why sample-level matching helps. An intuitive explanation may be required.  \n-\tFor the sample-level approach, it is computed within a mini-batch. How the performance is sensitive to batch-size?  \n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your valuable comments and questions. We are revising our rebuttal revision to address your concerns.\n> Weakness 1: It is hard to understand why sample-level matching helps. An intuitive explanation may be required.\n\nResponse to W1: Thanks for your comments. We assume that tokens at the same position may be related more closely in specific contexts and tasks. Consider the task of sentiment analysis in customer reviews for products. Here's a set of examples: \n\n Sentence 1: \"The camera quality is outstanding, making photography a delight.\"\n\n Sentence 2: \"The battery life is inadequate, making long-term use problematic.\"\n\n Sentence 3: \"The display clarity is excellent, making visual experience immersive.\"\n \nBy analyzing the relationships between samples in corresponding positions, we can discern several insights:\n\n* Contextual Relationship: We focus on the tokens in the 3th position, which are \"quality\", \"life\", and \"clarity\". These tokens, when considered in the context of product reviews, may offer valuable insights into different aspects that customers care about. By comparing these tokens, we could gain a deeper understanding of the particular features being praised or criticized in different reviews.\n\n* Subject Relationship: Despite the similarity in sentiment between sentences 1 and 3, the subjects are different (\"camera\" and \"display\"), which indicates that the positive sentiment is directed towards different entities. Analyzing the relationships between these tokens may be valuable for tasks that require understanding subjects and their associated sentiments or actions.\n\n* Sentiment Relationship: The tokens at the 5th position are \"outstanding\", \"inadequate\", and \"excellent\", which clearly relate to the sentiment expressed. Comparing these could indeed provide complementary knowledge for tasks like sentiment classification, where understanding the degree of positivity or negativity is crucial.\n\nIn our method, the relationships between tokens in the same position across different sentences are not meant to represent entire sentence meanings but to highlight specific aspects that may be relevant to certain tasks. By carefully choosing the context or domain, this approach can provide valuable insights and improve performance in those targeted areas.\n\n> Weakness 2: For the sample-level approach, it is computed within a mini-batch. How the performance is sensitive to batch-size?\n\nResponse to W2: Thank you for raising this concern. We evaluated our method with batch sizes of 8, 16, 32, and 64, observing the following performance on MNLI-m and QNLI:\n| Batch Size | MNLI-m | QNLI |\n|------------|--------|------|\n| 8          | 83.3   | 90.9 |\n| 16         | 83.7   | 91.2 |\n| 32         | 83.8   | 91.3 |\n| 64         | 83.9   | 91.3 |\n\nThe results indicate a robust performance across various batch sizes. However, notably, there is a moderate performance drop with extremely small batch sizes (i.e., less than 8).\n\n> Question1: line 292 says it is trained for 20 epochs which seems much longer than standard finetuning. Is it because the proposed approach is slow to converge?\n\nResponse to Q1:  Thanks for pointing out this. It is essential to clarify that while we mentioned 20 epochs in line 292, this does not imply a generally slow convergence for our proposed approach across all tasks. Specifically, the 20 epochs were adopted for the CoLA task, which can be more challenging and can benefit from extended finetuning. On the other hand, for tasks like QQP and MNLI, our method converged efficiently within just 3-5 epochs. Such variance in epoch settings, depending on the complexity of tasks, is aligned with practices observed in prior research as well [1][2]. We will emphasize this distinction more clearly in the revised paper to avoid any misunderstanding.\n\nReferences: \n\n[1] Wang, Wenhui, et al. \"Minilmv2: Multi-head self-attention relation distillation for compressing pretrained transformers.\" arXiv preprint arXiv:2012.15828 (2020).\n\n[2] Jiao, Xiaoqi, et al. \"Tinybert: Distilling bert for natural language understanding.\" arXiv preprint arXiv:1909.10351 (2019).\n\n> Question2: In Figure 4 (6.1), it is required to define B first before use in the function (token_relation_loss).\n\nResponse to Q2:  Thanks for pointing out this oversight in the supplementary material. In our implementation, B was defined within the forward method and was intended to be passed to the token_relation_loss function. We have rectified this in the revised paper."}, {"review_id": "opkbV0elEo", "reviewer": "Reviewer_NpzH", "summary": "This paper proposes a new knowledge distillation method named Feature Correlation Distillation (FCD) for compressing large pre-trained language models. The proposed method novelly uses token-level and sample-level relationship between teacher and student models and a Pearson linear correlation-based loss function to relax previous exact match restrictions from KL divergence and MSE loss. Extensive experiments show that the proposed method outperforms baseline methods on GLUE thanks to the proposed objectives. ", "questions": "Would it be possible to also demonstrate the efficacy of FCD theoretically or intuitively (e.g., through visualization)? The paper is already very good with empirical results, providing more evidence on why such a loss formation works well can be even better.", "limitations": "Not discussed, the authors should add a section.", "rating": 8, "confidence": 5, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "- The idea is novel and motivated by great intuitions, well explained in the paper.\n- The authors provided computation complexity analysis, which is very useful to understand the efficiency and complexity of the proposed method. This part is not usually presented in most papers. \n- The authors performed well-organized experiments and ablation studies, showcasing the benefits from FCD.", "weaknesses": "This paper really don't have obvious weaknesses. The only one might the lack of a limitation section.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your positive assessment and valuable suggestion.\n\n> Question 1: Would it be possible to also demonstrate the efficacy of FCD theoretically or intuitively (e.g., through visualization)? The paper is already very good with empirical results, providing more evidence on why such a loss formation works well can be even better.\n\nResponse to Q1: Thank you for your suggestion; it's indeed vital to complement empirical results with intuitive explanations. In our paper, we dedicated Section 3.4 to offer a theoretical rationale behind the effectiveness of the loss function, which is central to the FCD mechanism. Additionally, for a deeper dive, we've included in the supplementary materials both a detailed mathematical justification and a visualization illustrating the relational features before and after normalization (see Figure 5). Inspired by your feedback, we're now exploring the possibility of visualizing the loss and gradient landscapes. This will further elucidate how the loss function operates from an optimization perspective.\n"}, {"review_id": "tTDqAxoe9L", "reviewer": "Reviewer_XAJW", "summary": "This paper proposes a method for compressing pre-trained language models (PLMs) based on transformer architectures using feature correlation distillation (FCD). FCD models both token-level and sample-level relations between the teacher and student models, and uses a correlation-based loss function to relax the exact match constraint of traditional knowledge distillation methods. FCD achieves strong performance on the the GLUE benchmark with the teacher of the BERT and RoBERTa.", "questions": "Since this work has done GPT-2 distillation, why not also verify the effect on some generation tasks?", "limitations": "See the weakness and question section.", "rating": 5, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "A novel perspective to introduce sample-level information to help distillation. The design of learning objective is also interesting.", "weaknesses": "The token-level method is very similar to the attention-based method of Minilm. What is the essential difference? Also, this paper selects the last hidden before the prediction head as the feature, but is this really appropriate? Because the last hidden is too close to the output, a lot of information will be lost. Previous work (e.g., kNN-LM) mentioned that using the feature before the final layer’s FFN might be better.\n\nAlthough the sample-level idea sounds interesting, it doesn’t seem to make much sense. I’m not very clear why the sample-level information in a batch would be useful. If this is really useful, then not only in the distillation setting, but also in general training, sample-level information should be used (such as batch-norm methods). Otherwise, it is not very convincing to say that sample-level information is only useful in distillation.\n\nThe results of previous work reported in the paper are very inconsistent with the numbers reported in previous work. For example, when comparing with Minilm v2, the result numbers in Minilm v2 and those reported in this paper are quite different, which makes me concerned about whether the comparison is fair and whether the conclusions drawn from the comparison are convincing.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your valuable comments. We address your comments and questions in the following content.\n\n> Weakness 1: The token-level method is very similar to the attention-based method of Minilm. What is the essential difference? Also, this paper selects the last hidden before the prediction head as the feature, but is this really appropriate? Because the last hidden is too close to the output, a lot of information will be lost. Previous work (e.g., kNN-LM) mentioned that using the feature before the final layer’s FFN might be better.\n\nResponse to W1: Thank you for your valuable comments. We will address your concerns in two parts:\n\nDistinction from MiniLM: MiniLM’s attention is built upon the scaled dot-product interaction among queries, keys, and values, resulting in an attention matrix of size H x N x N (where H represents attention heads and N is the sequence length). This design inherently limits flexibility, especially when adjusting head counts. On the other hand, our approach forms a token-level relation matrix of size N x N via block features, offering more flexibility across models with varied attention heads.\n\nFeature Selection: We have added experiment to evaluate both pre and post-FFN features, and the results are summarized in the following table.\n\n| Model | MNLI-m | SST-2 |\n| ----------- | ----------- | ----------- |\n| (Teacher) BERT_Base   | 84.5 | 93.4\n| pre-FFN    | 83.6 | 92.5\n| post-FFN   | 83.8 | 92.8\n\nOur results indicate a slightly superior performance using post-FFN features. It's worth noting the divergence with kNN-LM; however, we're addressing different tasks, i.e., language modeling in kNN-LM versus our focus on distillation. A parallel observation was made with Vision Transformer (ViT) studies [1].\n\nReferences: \n\n[1] Yang, Zhendong, et al. \"Vitkd: Practical guidelines for vit feature knowledge distillation.\" arXiv preprint arXiv:2209.02432 (2022).\n\n> Weakness 2: Although the sample-level idea sounds interesting, it doesn’t seem to make much sense. I’m not very clear why the sample-level information in a batch would be useful. If this is really useful, then not only in the distillation setting, but also in general training, sample-level information should be used (such as batch-norm methods). Otherwise, it is not very convincing to say that sample-level information is only useful in distillation.\n\nResponse to W2: Your concerns about the sample-level concept's generalizability are valid. In essence, the sample-level approach examines inter-sample relationships within a batch. For instance, considering three customer reviews:\n\nS1: \"The camera quality is outstanding...\"\n\nS2: \"The battery life is inadequate...\"\n\nS3: \"The display clarity is excellent...\"\n\nBy comparing specific token positions across these samples, we gain valuable insights into the features and sentiments in various reviews. For instance, the tokens \"quality,\" \"life,\" and \"clarity\" in the third position provide a nuanced understanding of product attributes. Comparing tokens at the 5th position, i.e., \"outstanding\", \"inadequate\", and \"excellent\", could offer complementary insights for tasks such as sentiment classification, where the degree of positivity or negativity is crucial.\n\nWhile our study emphasized its utility in distillation, the potential of sample-level information isn't limited to this context. It has broader applications, as seen in certain computer vision tasks where batch relationships are explored during training, For example, [1] proposed BatchFormer module to model the relationships between different samples. Hence, its usefulness isn't exclusive to distillation, and future work can delve deeper into its applicability in standard training regimes.\n\nReferences:\n\n[1] Hou, Zhi, Baosheng Yu, and Dacheng Tao. \"Batchformer: Learning to explore sample relationships for robust representation learning.\" Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2022.\n\n> Weakness 3: The results of previous work reported in the paper are very inconsistent with the numbers reported in previous work. For example, when comparing with Minilm v2, the result numbers in Minilm v2 and those reported in this paper are quite different, which makes me concerned about whether the comparison is fair and whether the conclusions drawn from the comparison are convincing.\n\nResponse to W3: We acknowledge the inconsistency you've noticed. The primary reason for the variation is that our results were reported on the test sets of GLUE, while MiniLM v2's reported outcomes were from the development sets. To ensure fairness in comparison, we've re-implemented MiniLM v2 and evaluated it on the test sets.\n\n> Question 1: Since this work has done GPT-2 distillation, why not also verify the effect on some generation tasks?\n\nResponse to Q1: Thanks for your suggestion. While our primary focus was on distillation for classification tasks, exploring generative tasks would provide a more comprehensive understanding of our distillation approach's effectiveness. We did experiment with the WikiText-103 benchmark. Our model outperformed with a perplexity of 19.8 compared to distilgpt2's 21.1. This result suggests our method's potential applicability and benefits in the domain of generative tasks as well. We will add this result in the revised version to complement our evaluation."}, {"review_id": "yalJCeLvcF", "reviewer": "Reviewer_kWDa", "summary": "This paper proposes Feature Correlation Distillation (FCD), an approach for distilling pre-trained transformers. FCD involves a two-part distillation loss: (1) token-level and (2) sample-level, which helps to eliminate dependence on matching dimensionality/architectural details between the student and teacher and improves efficiency by constant factors. The paper also proposes replacing the KL-divergence or MSE based objective with a pearson correlation owing to its invariance to positive linear transformations. Experiments were run on GLUE with bert, roberta and distilgpt2, including comparing KL, MSE and correlation loss objectives and ablating the token and sample level losses.", "questions": "Would it be possible to expand the set of experiments to more tasks? Providing more varied empirical evidence for the approach could make the paper stronger.", "limitations": "NA", "rating": 7, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The paper proposes an interesting approach to distillation that relies on model internals while allowing flexibility in architectural choices. It proposes a more efficient distillation procedure which can be useful in resource constrained settings. The paper also makes an effort to empirically justify the chosen loss types and objectives via ablations.", "weaknesses": "GLUE tasks can have a large amount of variance due to randomness and due to small test sets, single-run scores may not be reliable (see e.g. https://arxiv.org/pdf/1904.09286.pdf: Section 4.2 note on variance, https://arxiv.org/abs/2010.06595 on small test sets). So, at the very least, averaging across random seeds and including errors bars in Figure 3 is important to understand if the results are robust. Since these experiments provide the only evidence for the claims and since the claims are largely empirical, adding robustness to the experiments seems crucial.\n\nNits:\nThe citation formatting appears to be off: try using \\citep and \\citet with natbib.\n\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your valuable comments. We address your comments and questions in the following content.\n\n> Weakness 1: GLUE tasks can have a large amount of variance due to randomness and due to small test sets, single-run scores may not be reliable (see e.g. https://arxiv.org/pdf/1904.09286.pdf: Section 4.2 note on variance, https://arxiv.org/abs/2010.06595 on small test sets). So, at the very least, averaging across random seeds and including errors bars in Figure 3 is important to understand if the results are robust. Since these experiments provide the only evidence for the claims and since the claims are largely empirical, adding robustness to the experiments seems crucial. Nits: The citation formatting appears to be off: try using \\citep and \\citet with natbib.\n\nResponse to W1: Thank you for your insightful feedback. We will address your concerns in two parts:\n\nRobustness in Experiments: We absolutely agree with with the points raised about the inherent variance in GLUE tasks and the potential implications for the reliability of single-run scores. All of our current results are already averaged across four different runs (as indicated in Table 1), and we acknowledge that this might not be sufficiently clear in Figure 3. In addition, in our revised paper, we will augment Figure 3 with error bars that capture the variability across these runs, providing a more comprehensive view of the robustness of our results.\n\nCitation Formatting: We're grateful for the feedback on our citation formatting. We'll make the necessary adjustments and ensure the use of \\citep and \\citet with natbib for consistent and standard citation representation in the updated version of the paper.\n\n> Question 1: Would it be possible to expand the set of experiments to more tasks? Providing more varied empirical evidence for the approach could make the paper stronger.\n\nResponse to Q1: Thank you for your valuable suggestion. Broadening the range of tasks evaluated is indeed beneficial for solidifying our empirical evidence. In addition to the tasks mentioned in the main paper, we have extended our experiments to SQuAD 1.1 and 2.0, which have been available in our supplementary materials. Moreover, based on suggestions from reviewer XAJW, we've tested our approach on the WikiText-103 benchmark for generation task. Our model showcased an improved perplexity of 19.8 against distilgpt2's 21.1. This finding underscores the versatility of our methodology, even beyond classification tasks. We're open to expand our empirical evaluation further, based on the tasks recommended by the reviewers, to cement our approach's validity across diverse NLP applications. Your guidance in this regard is invaluable.\n\n| Model | SQuAD 1.1 | SQuAD 2.0 |\n| ------ | ------- | ------ |\n| (Teacher) BERT_Base   | 88.7 | 78.8\n| DistilBERT   | 86.2 | 69.5\n| TinyBERT  | 87.5 | 77.7\n| Ours  | 88.2 | 78.4\n\n| Model | WikiText-103 \n| ------ | ------- |\n| (Teacher) GPT2   | 16.3\n| distilgpt2   | 21.1 \n| Ours   | 19.8"}, {"review_id": "jJdcIlh5kd", "reviewer": "Reviewer_R5h8", "summary": "This paper introduces a new technique for compression the PLM. Within the context of knowledge distillation, the authors introduce a new type of relation distillation, which focuses on modeling relations between token features at both the token-level and sentence-level. These relations are then utilized as an objective function for the teacher-student distillation process. Additionally, the authors propose a correlation-based distillation loss as a replacement for the previous distillation loss function that relied on exact match as the target. Experimental results show that FCD achieves significant improvements over the previous methods.", "questions": "1. Upon reviewing the results in Table 1, I noticed that none of the reported results on the test set on GLUE benchmark, including BERT_base, are consistent with those presented in their original paper. Some results are higher than the previous paper, while others are lower. This raises the question of whether all the baseline results were reproduced or if there were modifications to the student structure or different settings employed. Could you clarify why there are changes to the baselines results?\n2. The baseline \"BERT_SMALL6\" involves dropping some layers and fine-tuning on the downstream datasets without knowledge distillation (KD), which can be interpreted as the L_g loss in Eq.10. However, I noticed a significant discrepancy between the reported SST-2 result of 90.7% and the result shown in Figure 3.c's ablation experiments, where using only L_g on SST-2 yields approximately 86%. Could you explain the reason behind this substantial gap between the two results?\n3. The motivation for FCD is to address the size misalignment problem; however, in the conducted experiments, both the teacher and student models maintain the same structure, except for dropping certain layers. Consequently, there doesn't appear to be a misalignment issue in dimension and the number of attention aheads. Could you provide some experiments on the student model will less attention head and also reduced dimension?\n4. Can you provide with some explanation why the sample-level relation loss can aid in distillation?\n5. Have you attempted to build FCD upon released distilled models with pre-training, such as TinyBERT (e.g., TinyBERT_General_\n*L_*D)? It will be interesting to see combining these two techniques together to boost the performance of the distilled model.", "limitations": "N/A", "rating": 4, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "1. A clear motivation for the method: resolving the mismatch in size during alignment between the teacher and the student, such as the number of heads or the dimension of the hidden state. The authors propose modeling both token-level and sample-level relationships, which remains consistent between the teacher and the student models, regardless of their model design. This approach can help to minimize the impact of alignment mismatch.\n2. The method offers a simple and effective solution for distillation, supported by proof that demonstrates the relationship between PLC, KLD, and MSE.\n3. The presentation of method is clear and easily comprehensible.\n4. It is remarkable to observe that FCD, a task-specific distillation approach, surpasses the performance of task-agnostic KD with pre-training, like TinyBERT and MiniLMv2.", "weaknesses": "1. The idea of sample-level relation loss appears unreasonable to me. The approach of comparing the j-th token in each sentence and using the dot-product similarity to represent sentence-level relationship seems unreasonable. It raises doubts about whether the relationships between tokens in the same position, but in different sentences, truly reflect the relationships between the sentences and are directly comparable. For instance, consider the following example:\nS1: For simplicity and clarity, our paper focuses...\nS2: For your safety, please ensure that you are wearing protective...\nIf we focus on the first token, they are the same, yet the sentences have completely different meanings.\n\n2. I have some doubts regarding the experimental results and would appreciate further clarification from the authors to determine if any potential weaknesses exist. Please refer to questions 1, 2, and 3.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your valuable comments and efforts in reviewing our paper. We address your comments and questions in the following content.\n\n> Weakness 1: The concept of sample-level relation loss seems unreasonable. Comparing the j-th token in each sentence to determine sentence-level relationships is questionable. For example, sentences \"S1\" and \"S2\" both start with the same token but convey different meanings.\n\nResponse to W1: Indeed, the example you provided highlights that individual token similarity may not always reflect the holistic meaning of sentences. However, our proposed sample-level relation loss, as described in Eq.9, is not solely based on one isolated token comparison. Instead, it focuses on an aggregate assessment across all tokens in a sequence. To put it simply, our approach isn't just about direct token-to-token comparison but rather the interaction and aggregation of these tokens across an entire sentence.\nConsider the following customer review examples:\n\nS1: \"The camera quality is outstanding...\"\n\nS2: \"The battery life is inadequate...\"\n\nS3: \"The display clarity is excellent...\"\n\nIf we assess token positions across these sentences, we observe patterns associated with product attributes (e.g., tokens in the third position) and sentiment (e.g., tokens in the fifth position). This suggests that there's a relationship not only in the individual tokens but in the context they create.  Thus, we define sentence-level relationships with token-wise relation matrices across all positions and model them in a nuanced and meaningful manner.\n\n> Question 1: In Table 1, the results for the GLUE benchmark, including BERT_base, differ from the original paper. Were the baseline results reproduced as is, or were there changes to the student structure or settings? Please explain the discrepancies.\n\nResponse to Q1: Thank you for noting the discrepancies in Table 1. We used public pre-trained models for BERT_base, DistilBERT, and TinyBERT. For MiniLM v1 and v2, we re-implemented them due to differences in their publicly available structures and the absence of test set results. All models were fine-tuned using consistent settings. The variations from original papers might be due to training randomness and specific training details, like the omission of data augmentation used in the TinyBert.\n\n> Question 2: The baseline \"BERT_SMALL6\" involves dropping some layers and fine-tuning on the downstream datasets without KD, which can be interpreted as the L_g loss in Eq.10. However, I noticed a significant discrepancy between the reported SST-2 result of 90.7% and the result shown in Figure 3.c's ablation experiments, where using only L_g on SST-2 yields approximately 86%. Could you explain the reason behind this substantial gap between the two results?\n\nResponse to Q2: We appreciate your careful attention to this discrepancy in our paper. The difference arises from the configurations in Figure 3.c where we applied both L_g and L_s, in contrast to BERT_SMALL6 which only utilized L_g. Upon revisiting our setup, we identified that the beta weight factor was mistakenly set too high, causing an imbalance between the losses L_s and L_g.This misconfiguration resulted in the observed performance gap. After correcting this and re-running our experiments, we achieved an average SST-2 score of 91.9% over four runs. We will amend Figure 3.c in the revised version to reflect this.\n\n> Question 3: Could you provide some experiments on the student model will less attention head and also reduced dimension?\n\nResponse to Q3: Thank you for your valuable suggestion. We conducted experiments on student model with reduced attention heads and dimensions. Specifically  we set student models to be 6-layer, 6-head, with a hidden size of 384. Here are the results:\n\n| Model | MNLI-m | SST-2 |\n| ------ | ----- | --------|\n| (Teacher) BERT_Base   | 84.5 | 93.4\n| TinyBert_6H_6L_384    | 79.8 | 88.2\n| MiniLMv2_6H_6L_384    | 81.2 | 89.6\n| FCD_6H_6L_384   | 81.8 | 90.3\n\nFCD's consistent outperformance highlights its capability in alleviating this misalignment problem, which aligns with the motivation of our approach. While the performance of these compact models doesn't reach our original paper's results, potentially due to lack of pre-trained initialization, integrating strategies like attention head pruning could further optimize results. We will incorporate these results in the updated version.\n\n> Question 4: Can you provide with some explanation why the sample-level relation loss can aid in distillation?\n\nResponse to Q4: Please see our response to W1. Let us know if it does not answer your question.\n\n> Question 5: Have you attempted to build FCD upon released distilled models with pre-training, such as TinyBERT (e.g., TinyBERT_General_ *L_*D)? It will be interesting to see combining these two techniques together to boost the performance of the distilled model.\n\nResponse to Q5: Thank you for posing this insightful question.  We explored the idea by building FCD upon pre-trained TinyBERT models and subsequently fine-tuned them on downstream tasks without data augmentation. Here's a summary of the results:\n\n| Model | RTE | SST-B |\n| ----------- | ----------- | ----------- |\n| (Teacher) BertBase   | 66.8 | 85.2\n| TinyBert_6L_768    | 70.0 | 83.9\n| TinyBert_4L_312    | 64.1 | 80.4\n| (FCD) TinyBert_6L_768   | 71.9 | 85.4\n| (FCD) TinyBert_4L_312   | 67.3 | 81.1\n\nOur experiment shows that FCD combined with general distillation not only outperforms the regular TinyBERT model on RTE and SST-B tasks but also those reported in our original paper, which suggests that initializing the student model with general distillation can provide a robust foundation for further fine-tuning with our FCD method. This observation aligns well with the findings presented in other recent work [1].\n\nReferences:\n\n[1] Lu, Chengqiang, et al. \"Knowledge Distillation of Transformer-based Language Models Revisited.\" arXiv preprint arXiv:2206.14366 (2022)."}]

Author rebuttal:
We thank all the reviewers for their valuable feedback. The detailed responses to the reviewers’ comments will be replied directly to each reviewer.

Meta-review: this paper focuses on task-specific knowledge distillation and proposed a new method the author call Feature Correlation Distillation (FCD) for overcoming differences in architecture and dimensions between the teacher and student. The authors proposed method is based on token-level and sample-level relationships between teacher and student. Also, a pearson correlation loss is used instead on the common KL divergence or MSE, with a theoretical analysis for justifying its benfits. The reported empirical results on GLUE, together with additional results from the rebuttal (that should be added to the paper) demonstrate good performance.

Some of the concerns that were brought up were around the noisiness of GLUE tasks and slight differences in scores with original papers due to retraining. I strongly recommend to include additional results such as the ones on SQuAD reported in the rebuttal, and also to report some confidence intervals (perhaps with bootstrapping if multiple runs are too expensive). Also, FCD implicitly assumes some relation across tokens from different samples in the same position which is not entirely intuitive. The author propose some hand-wavy justifications, but I strongly recommend at least discussing this as a limitation in the paper since this assumption might not be true for some target data distributions.
Decision: Accept (poster)