AD-KD: Attribution-Driven Knowledge Distillation for Language Model Compression
Year: 2023

Abstract:
Knowledge distillation has attracted a great deal of interest recently to compress pre-trained language models. However, existing knowledge distillation methods suffer from two limitations. First, the student model simply imitates the teacher’s behavior while ignoring the underlying reasoning. Second, these methods usually focus on the transfer of sophisticated model-specific knowledge but overlook dataspecific knowledge. In this paper, we present a novel attribution-driven knowledge distillation approach, which explores the token-level rationale behind the teacher model based on Integrated Gradients (IG) and transfers attribution knowledge to the student model. To enhance the knowledge transfer of model reasoning and generalization, we further explore multi-view attribution distillation on all potential decisions of the teacher. Comprehensive experiments are conducted with BERT on the GLUE benchmark. The experimental results demonstrate the superior performance of our approach to several state-of-the-art methods.

heading: 1 Introduction
text: Transformer-based pre-trained language models (PLMs), such as BERT (Devlin et al., 2019) and RoBERTa (Liu et al., 2019), have aroused widespread interest among Natural Language Processing (NLP) researchers in recent years. These language models are first pre-trained on large-scale unlabeled corpora to learn the general representation of language, and then fine-tuned on specific downstream tasks to effectively transfer the general knowledge to target domains. This pre-training and fine-tuning paradigm leads to state-of-the-art performances in various NLP tasks such as natural language understanding. However, with the rapid growth of the model scale, the deployment of large-scale PLMs becomes challenging, especially in low-resource scenarios. To this end, a variety
∗Corresponding author.
of model compression techniques have been developed. Among them, knowledge distillation (KD) (Hinton et al., 2015) is a newly emerging technology that aims to obtain a small student model by distilling knowledge from a large teacher model and achieve comparable performance.
Existing knowledge distillation methods can be divided into three categories, namely responsebased, feature-based, and relation-based (Gou et al., 2021). While response-based methods (Turc et al., 2019) directly distill the final output, e.g. probability distribution, from the top of the teacher, feature-based (Sun et al., 2019) and relation-based methods (Liu et al., 2022) try to align the features from intermediate layers of teacher and student models and minimize the difference. To transfer comprehensive knowledge from the teacher, a common practice is to combine response-based methods with the other two (Park et al., 2021). However, due to the capacity gap between the teacher and the student, feature-based and relation-based methods may not necessarily bring improvement to response-based methods (Liang et al., 2022). To sum up, existing knowledge distillation methods have two limitations. First, they mainly focus on understanding what the teacher’s behavior is, instead of why the teacher behaves like this, hinder-
8449
ing the reasoning and generalization ability of the student model. Second, they pay more attention to distilling sophisticated model-specific knowledge from intermediate layers but neglect data-specific knowledge, which may contain valuable rationale information to understand how the teacher model arrives at a prediction.
To address the above limitations, in this paper we propose a novel Attribution-Driven Knowledge Distillation (AD-KD) approach that transfers attribution-based knowledge from the teacher to the student. As shown in Figure 1, the attribution information reflects the importance of different tokens towards the prediction, which contains reasoning knowledge of the model and can be complementary to the soft-label knowledge. By transferring such attribution knowledge, the student is allowed to learn the token-level rationale behind the teacher’s behavior and thus generalizes better. Specifically, we utilize Integrated Gradients (IG) (Sundararajan et al., 2017), a well-established gradient-based attribution method, to calculate the importance score of each input token. To reduce the influence of trivial dimensions in the teacher’s input embeddings, we further adopt the top-K strategy to filter out dimensions with low attribution scores. The remaining attribution scores are aggregated and normalized to denote the importance of individual tokens. Moreover, we extract the attribution knowledge for all possible predictions rather than just the prediction with the highest probability. By transferring the multi-view attribution knowledge, the student learns a more comprehensive understanding of the teacher’s soft-label distribution.
Extensive experiments are conducted with BERT (Devlin et al., 2019) on the GLUE benchmark (Wang et al., 2018). The experimental results demonstrate the effectiveness and superiority of our approach over several state-of-the-art baselines. Furthermore, we show that attribution knowledge from different layers contains different information, while the input layer contains the most prominent attribution knowledge for distillation. To summarize, the main contributions are threefold. First, we propose a novel attribution-driven knowledge distillation framework for language model compression that effectively transfers attribution knowledge from the teacher to the student. Second, we extract multi-view attribution knowledge based on model predictions to learn comprehensive reasoning knowledge. Third, we systematically validate
AD-KD on the GLUE benchmark and show its superior performance over state-of-the-art baselines.

heading: 2 Related Work

heading: 2.1 Knowledge Distillation
text: Knowledge distillation methods can be divided into three categories, namely response-based, featurebased and relation-based KD (Gou et al., 2021). Response-based KD was first proposed by Hinton et al. (2015), where the final output is adopted to transfer the label knowledge. Sanh et al. (2019) and Turc et al. (2019) applied this idea to BERT and yielded smaller models with minor performance drops. Recently, feature-based and relation-based distillation methods have drawn a lot of attention, which transfer knowledge contained in the intermediate layers to the student. For feature-based methods, Sun et al. (2019) first regarded the hidden representations of the [CLS] token as hints to extract sentence-level features from the teacher. Jiao et al. (2020) and Sun et al. (2020b) further matched the hidden representations of all tokens between teacher and student models. Sun et al. (2020a) proposed contrastive distillation on intermediate representations. As for relation-based methods, Park et al. (2021) proposed CKD which adopts pair-wise distance and triple-wise angle to model the sophisticated relations among token representations from both horizontal and vertical directions. Based on CKD, Liu et al. (2022) further extracted structural relations from multi-granularity representations and distilled this kind of well-organized multi-granularity structural knowledge hierarchically across layers. Wang et al. (2020, 2021) generalized the conventional query-key attention to query-query attention, key-key attention, and valuevalue attention. Different from these methods, we investigate knowledge distillation from the attribution perspective, which reveals the teacher’s reasoning behavior and can be used to transfer comprehensive data-specific knowledge. More details about the differences between existing methods and ours are discussed in Appendix B.

heading: 2.2 Attribution
text: Attribution analysis (Baehrens et al., 2010; Ancona et al., 2018) aims at assigning importance scores to intermediate or input features of a network. Occlusion-based methods (Zeiler and Fergus, 2014) compute the importance score of each feature by erasing that feature and measuring the
difference between new output and the original output. However, occlusion-based methods need to forward pass the model once for each feature, leading to low computational efficiency. To address this issue, gradient-based methods (Li et al., 2016; Ding et al., 2019; Brunner et al., 2020; Sundararajan et al., 2017) exploit the gradient information of features to approximate occlusionbased methods, which only require a single forward process. Similarly, propagation-based methods (Bach et al., 2015; Shrikumar et al., 2017) modify the back-propagation rules to redistribute the model output among the target features along the back-propagation path. Perturbation-based methods (Guan et al., 2019; Schulz et al., 2020; De Cao et al., 2020) add noise to features to examine their importance for model predictions. Attribution has been adopted in model compression techniques such as pruning (Michel et al., 2019) and adaptive inference (Modarressi et al., 2022) but has not been explored in knowledge distillation. In this work, we take the initiative to investigate the effect of attribution in knowledge distillation.

heading: 3 Methodology

heading: 3.1 Preliminary
text: Integrated Gradients (Sundararajan et al., 2017) is a theoretically tenable method to attribute the prediction of a deep network to its input or intermediate features. Formally, given a feature x = [x1, x2, ..., xn] ∈ Rn with a baseline feature x′ = [x′1, x ′ 2, ..., x ′ n] ∈ Rn, and the model function F (.), IG leverages integral to represent the difference between F (x) and F (x′) by selecting a straight line path from x′ to x as the integral path:
F (x)− F (x′) = n∑
i=1
IGi(F,x) =
n∑
i=1
[(xi − x′i)× ∫ 1
α=0
∂F (x′ + α× (x− x′)) ∂xi dα].
(1)
In practice, continual integral can be approximated by discrete summation:
IGapproxi (F,x) =
(xi − x′i)× m∑
k=1
∂F (x′ + k m × (x− x′)) ∂xi × 1 m , (2)
where m is the number of summation steps (a bigger m usually results in better approximation). Intuitively, the magnitude of integrated gradient indicates its importance while its sign illustrates the positive or negative effect on the target output.
In this paper, we focus on Transformer-based architecture and attribute the model prediction to input features. With slight abuse of notation, we denote the input sequence as x = [x1, x2, ..., xn], where n is the sequence length and each xi represents a token. Transformer first converts the token sequence to d-dimensional embedding sequence E = [e1, e2, ..., en] ∈ Rn×d through the embedding layer. And then the contextualized representations H = Transformer(E) ∈ Rn×d are obtained after several layers of Transformer blocks. Finally, a task-specific head is applied on H to get the final output P = [P1, P2, ..., PC ] ∈ RC , which is typically a probability distribution. Denote the mapping function E → Pc as F c(.), where c represents the label of interest. In this case, our attribution map is computed on each individual dimension of each input embedding, which is denoted as eij :
IGapproxij (F c,E) =
(eij − e′ij)× m∑
k=1
∂F c(E′ + k m × (E−E′)) ∂eij × 1 m . (3)
In the implementation, we stack n [PAD] token embeddings as baseline features E′ since they usually have no influence on the model prediction.

heading: 3.2 AD-KD
text: In this section, we elaborate on our proposed Attribution-Driven Knowledge Distillation (ADKD), including attribution maps and attribution distillation. The overall framework of AD-KD is illustrated in Figure 2.

heading: 3.2.1 Attribution Maps
text: The attribution scores of a language model reflect the importance of different tokens towards the prediction, which contains valuable data-specific reasoning knowledge. The scores are computed among different tokens at different dimensions of a given model, using IG defined in Section 3.1. In this work, we do not take the sign into consideration, since the scores at different dimensions of the same token embedding would cannibalize each other when combining them into a token-level attribution score. This observation is consistent with the findings in (Atanasova et al., 2020).
When calculating the attribution scores, we observed that there exist certain dimensions whose attribution scores remain relatively low across different tokens. The attribution scores from these dimensions minimize the difference between important and unimportant tokens, which can be regarded
as noises. For better illustration, Figure 3 shows an example of sentence “seem weird and distanced” whose annotation is negative sentiment. It is clear that “weird” and “distance” are the keywords that contribute most to the prediction, whereas a proportion of dimensions of them present low attribution scores. To alleviate the influence of noisy dimensions in the input embeddings, we simply choose the top-K dimensions with high attribution scores and filter out dimensions with low attribution scores. Formally, the attribution score of token xi with respect to the label c in the teacher model can be calculated as:
at,ci = ∥TopK(IGapproxi (F t,c,Et))∥2, (4)
where the superscript t denotes the teacher model. Therefore, the attribution map of the teacher consists of a sequence of attribution scores:
at,c = [at,c1 , a t,c 2 , ..., a t,c n ]. (5)
For the student, the extraction of attribution map is similar except that we consider all dimensions for two reasons. First, it reduces the difficulty of training. Second, the student is allowed to learn from the noiseless attribution map of the teacher.
as,ci = ∥IGapproxi (F s,c,Es)∥2, as,c = [as,c1 , a s,c 2 , ..., a s,c n ].
(6)
Considering that the teacher can make multiple decisions, each of which is associated with a probability, we further propose to extract multi-view attribution knowledge. Specifically, we extract the
attribution maps for all possible predictions of the model rather than a single prediction, e.g., the prediction with the maximum probability or the prediction corresponding to the ground-truth label. By transferring the multi-view attribution knowledge, the student can capture a more comprehensive understanding of the teacher’s soft-label distribution. The multi-view attribution maps are defined as:
At =∥Cc=1 at,c, As =∥Cc=1 as,c, (7)
where ∥ is the concatenation operation.

heading: 3.2.2 Attribution Distillation
text: Given the multi-view attribution maps, a straightforward strategy to transfer the knowledge is to directly minimize the difference between the two sets of maps in teacher and student models, with distance metrics like L2 distance (MSE):
∥At −As∥2. (8)
However, one obvious shortcoming with this approach is that there may exist a magnitude gap between the attribution scores in teacher and student models at the early phase of distillation, since the teacher is already well-trained while the student has little attribution knowledge. Under this circumstance, the student is likely to fall into a local optimum. To enable smooth knowledge distillation, we normalize the attribution maps before minimizing the difference. Concretely, we first transform the single-view attribution maps into unit vectors:
ãt,c = at,c
∥at,c∥2 , ãs,c =
as,c
∥as,c∥2 . (9)
Then we reformulate the normalized multi-view attribution maps in Eq. (7) as:
Ãt =∥Cc=1 ãt,c, Ãs =∥Cc=1 ãs,c. (10)
The normalized attribution maps only preserve the information of relative importance among tokens regardless of their absolute importance, which we believe is the crucial knowledge to transfer. Finally, we define the attribution distillation loss as:
Lattr = ∥Ãt − Ãs∥2. (11)

heading: 3.2.3 Overall Objective
text: We combine the original cross-entropy loss between the output of the student and the groundtruth label, the response-based loss (on the logits) (Hinton et al., 2015), and the proposed attributiondriven distillation loss to train the student model. The overall objective is defined as:
L = (1− α)Lce + αLlogit + βLattr, (12)
where Lce = −logσ(zs)[y] is the cross-entropy loss and Llogit=KL(σ( z t τ )∥σ( z s
τ )) is the loss on the output logits. And, α and β are two hyperparameters, σ is the softmax function, y is the ground-truth label, τ is the temperature, and zt and zs are the output logits of the teacher and student models, respectively. KL(·) denotes the KL-divergence.

heading: 4 Experimental Settings

heading: 4.1 Datasets
text: We evaluate our method on eight tasks of the GLUE benchmark (Wang et al., 2018), including CoLA (Warstadt et al., 2019), MNLI (Williams et al., 2018), SST-2 (Socher et al., 2013), QNLI (Rajpurkar et al., 2016), MRPC (Dolan and Brockett, 2005), QQP (Chen et al., 2018), RTE (Bentivogli et al., 2009) and STS-B (Cer et al., 2017). The details of these datasets are introduced in Appendix
A.1. For evaluation metrics, we follow previous works (Park et al., 2021; Liu et al., 2022) and report accuracy on MNLI, SST-2, QNLI, QQP and RTE, F1 score on MRPC, Matthews correlation coefficient on CoLA, and Spearman’s rank correlation coefficient on STS-B.

heading: 4.2 Baseline Methods
text: We compare AD-KD with response-based KD methods and several state-of-the-art feature-based and relation-based KD methods. Response-based baselines include Vanilla KD (Hinton et al., 2015) and PD (Turc et al., 2019). Feature-based and relation-based baselines include PKD (Sun et al., 2019) which distills the hidden representations, TinyBERT (Jiao et al., 2020) which distills the selfattention matrices, and CKD (Park et al., 2021) and MGSKD (Liu et al., 2022) which distill the relation between hidden representations. For a fair comparison, MiniLM (Wang et al., 2020, 2021) and MobileBERT (Sun et al., 2020b) are not presented due to their two-stage distillation settings which involve both task-agnostic and task-specific distillation. Our AD-KD focuses on task-specific distillation and does not augment the training sets. Moreover, MGSKD (Liu et al., 2022) only reports results on a 4-layer BERT student model which is different from other baselines. To ensure a fair comparison, we re-implemented MGSKD using their released code to obtain a 6-layer student model. The original MGSKD approach also relies on spanlevel information that is extracted from external knowledge sources, which is not publicly available nor included in other baselines. Therefore, we did not use this external knowledge in our reimplementation of MGSKD.

heading: 4.3 Implementation Details
text: Our code is implemented in Pytorch with the Transformers package (Wolf et al., 2020). We fine-
tune BERTbase as the teacher model, and utilize a smaller BERT released by Turc et al. (2019) with 6 Transformer layers, 768 hidden neurons and 12 attention heads to instantiate the student model following Park et al. (2021). We search for the optimal learning rate in {2e-5, 3e-5, 4e-5, 5e-5}, α in {0.8, 0.9, 1.0} and temperature τ in {1, 2, 3, 4}. For the hyperparameter β, we tune within {1, 10, 50, 100}. For the IG steps m described in Section 3.1, we adopt m = 1 in the main results due to the huge computational overhead. Part of results with m varying from 1 to 8 are reported in Section 5.4. K is empirically searched within {384, 512, 640, 700, 734, 768}. Results with different values of K are also reported. The detailed hyperparameter settings and training cost are provided in Appendix A.2. Our code is available at https://github.com/brucewsy/AD-KD.

heading: 5 Results and Analysis

heading: 5.1 Main Results
text: The main results are presented in Table 1. It can be seen that AD-KD outperforms all baselines on most of the datasets. Specifically, AD-KD yields an average improvement of 1.0 and 1.9 points over CKD and MGSKD respectively on development sets, and another average improvement of 0.9 points over MGSKD on test sets. Note that other feature-
based and relation-based KD methods even underperform vanilla KD, indicating the difficulty of aligning the teacher and the student at intermediate layers. In contrast, AD-KD distills the attribution knowledge from a global perspective which is more data-specific and shows significant improvement over vanilla KD. We provide two cases in Appendix C.3 to intuitively demonstrate the strength of ADKD. We also observe that AD-KD does not show a satisfying performance on SST-2. We believe the reason is that the sentences in SST-2 are much shorter than those in other datasets, and in this case, the student is likely to already capture the attribution knowledge implicitly from the soft-labels of the teacher (Zhang et al., 2022).

heading: 5.2 Ablation Study
text: Impact of Loss Terms To analyze the impact of different loss terms, we conduct ablation experiments on three variants of AD-KD: (1) AD-KD without attribution distillation (i.e., vanilla KD), (2) AD-KD without the original cross-entropy loss, and (3) AD-KD without logit distillation. As reported in Table 2, again we observe an obvious performance drop after removing the attribution distillation. We also note that removing either the conventional cross-entropy loss or logit distillation loss causes noticeable performance degradation, suggesting both of them contribute to the improve-
ment of AD-KD. Nevertheless, our attribution distillation contributes most to the performance of AD-KD, showing that data-specific reasoning information is crucial in knowledge distillation. Multi-view Attribution In AD-KD, the student learns the attribution knowledge from a variety of possible outputs to get a better understanding of the teacher. Here we study how the number of attribution views affects the final results. Experiments are conducted on MNLI which is a multi-classification task including three labels: entailment, contradiction, and neutral. We make a comparison between multi-view attribution and single-view attribution w.r.t. each candidate label respectively. The results are shown in Figure 4, from which we note that each of the single-view attributions plays a positive role and is superior to vanilla KD. Moreover, combining all attribution views yields further performance improvement, demonstrating that multiview attribution is more preferable for distillation. Student Model Size To investigate whether ADKD can boost the performance across different sizes of student, we further compare AD-KD with vanilla KD on MRPC and QNLI under various student scales provided by Turc et al. (2019). As observed in Figure 5, AD-KD consistently outperforms vanilla KD, which validates the effectiveness and stability of our approach.
5.3 Impact of Top-K
Recall that in order to eliminate the interference of noisy dimension, AD-KD adopts the top-K ap-
proach on the input embeddings of the teacher to filter out the dimensions with relatively low attribution scores. In this section, we conduct in-depth analysis on the impact of K. We conduct experiments on STS-B and QNLI, and plot the results with different values of K in Figure 6. As illustrated in the figure, the performance on the small dataset STS-B (7k) first improves as K increases and then slightly degrades after K exceeds 600. However, the performance on the larger dataset QNLI (108k) improves almost monotonically with the increasing of K. We conjecture that choosing a suitable K is beneficial on small datasets since there are probably more noisy dimensions in the input embeddings of the teacher, while preserving all dimensions may be preferable on larger datasets.

heading: 5.4 Impact of IG Steps
text: In our experiments, the IG steps m are set to 1 by default when extracting the attribution maps. In this section, we provide more results with different values of m in Figure 7 to understand its impact on distillation. We observe that as m increases,
the performance of AD-KD fluctuates in a certain range. Although it is possible to find a point that surpasses our default setting and even the teacher, identifying the optimal value of m for each task is costly since a large m causes huge computational overhead. In contrast, m=1 achieves better tradeoff between performance and computational cost.

heading: 5.5 Attribution Distillation Layer
text: Apart from the attribution knowledge of input layer, the attribution knowledge of intermediate layers can also be transferred during distillation. To confirm the motivation that the former is better than the latter, we conduct experiments on MRPC and QNLI with different attribution layers. Specifically, we choose the first layer and the penultimate layer for comparison. Besides, we also try a uniform strategy which is widely adopted as the mapping function between the teacher and the student layers (Jiao et al., 2020; Park et al., 2021; Liu et al., 2022). From the results shown in Table 3, we see that uniform mapping strategy performs best among
intermediate layer methods. However, neither of these intermediate layers outperforms input layer, indicating that the attribution knowledge of intermediate layers is more model-specific and difficult to transfer. In addition, distilling the knowledge jointly from the input and the intermediate layers does not improve the performance.

heading: 5.6 Impact of α and β
text: For the training objective of AD-KD, we introduce α and β to balance the original cross-entropy loss, logit distillation loss, and attribution distillation loss. To investigate their impact on model performance, we show the results of different values of α and β on MRPC and QNLI in Figure 8, where we fix one while altering the other. We observe a unified trend across different tasks that when α is small, the student does not perform well due to the lack of response-based knowledge of the teacher, and when α is around 0.9, the student performs best. Therefore, we select α close to 1. We also observe from the figure that as β increases, the performance first keeps improving and reaches the peak, then it starts to decline. Unlike α, however, the optimal value of β varies with different tasks, indicating that β is more sensitive to the task compared to α. More discussion of β are given in Appendix C.2.

heading: 6 Conclusion
text: In this paper, we propose AD-KD, a novel knowledge distillation framework for language model compression. Unlike other distillation methods, AD-KD investigates the model knowledge from the perspective of input attribution, which is vital yet easy to transfer between the teacher and the
student. Moreover, top-K method is adopted to obtain noiseless attribution maps among input tokens, and multi-view attribution is conducted for a more comprehensive distillation. To our knowledge, this is the first work that incorporates attribution into knowledge distillation. Extensive experiments including ablation studies are carried out to show the effectiveness of AD-KD and its components. With the recent emergence of large language models (LLMs), gradient-based attribution methods are infeasible due to the unavailable parameters. However, the idea of AD-KD can still be potentially extended to these black-box models by using occlusion-based attribution or using chainof-thoughts (Wei et al., 2022) as the rationale for distillation. We will leave it to future work.

heading: Acknowledgements
text: This work was supported by the National Natural Science Foundation of China (No. 62176270), the Guangdong Basic and Applied Basic Research Foundation (No. 2023A1515012832), and the Program for Guangdong Introducing Innovative and Entrepreneurial Teams (No. 2017ZT07X355).
Limitations
This work introduces the general idea of incorporating attribution into knowledge distillation, and there are three potential limitations. First, although AD-KD chooses Integrated Gradients for attribution, there are actually other attribution methods (Janizek et al., 2021; Sikdar et al., 2021) which can also be fitted in our framework. The question of whether these methods perform better than Integrated Gradients when combined with knowledge distillation is still unclear. Second, we conduct experiments on BERT of different scales and have not yet validated the effectiveness of AD-KD on other model structures. Third, while we only perform task-specific knowledge distillation in our experiments, applying AD-KD to task-agnostic knowledge distillation is also worth investigating.
Ethics Statement
Our work will not cause ethical issues and the datasets used in this paper are publicly available.

heading: A Experimental Details
text: A.1 Details of Datasets
We evaluate AD-KD on eight tasks of GLUE benchmark (Wang et al., 2018). Specifically, there are two single-sentence tasks: CoLA (Warstadt et al., 2019) which aims to predict if the given sentence is grammatically correct, and SST-2 (Socher et al., 2013) which aims to predict the sentiment of the given sentence; two paraphrase tasks: MRPC (Dolan and Brockett, 2005) which aims to predict if two given sentences are semantically equivalent, and QQP (Chen et al., 2018) which is similar to MRPC; three inference tasks which aim to predict if the premise entails the hypothesis: MNLI (Williams et al., 2018), QNLI (Rajpurkar et al., 2016), and RTE (Bentivogli et al., 2009); and one similarity task: STS-B (Cer et al., 2017) which aims to predict a continual score measuring the semantic similarity between a pair of sentences. The statistics of these datasets are shown in Table 4.
A.2 Hyperparameter Settings
We run all experiments on GeForce RTX 2080 Ti GPUs. Table 5 presents the hyperparameter settings and training costs of AD-KD on GLUE tasks. Generally, AD-KD runs 1.2 to 3 times slower compared to vanilla KD on different tasks, due to the extra back-propagation. However, all students obtained by different distillation methods have the same inference speed.

heading: B More Discussion
text: In this section, we discuss the difference between distilling the attribution maps and distilling the attention matrices. In a sense, attention matrices are similar to attribution maps since they both reflect the contribution that each input token makes on a
model prediction to some extent (Bastings and Filippova, 2020; Xu et al., 2020). However, there are several drawbacks when it comes to distillation. On one hand, attention correlates well with attribution locally in specific layers and heads but not globally, indicating that attention maps are inadequate to draw conclusions that refer to the input of the model (Pascual et al., 2021). In other words, attention matrices are more like model-specific knowledge that are probably challenging for the student to learn due to the layer mapping issue, especially when the student has much fewer parameters than the teacher. On the other hand, some works point out that by adversarial training, alternative attention weights can be found whereas the prediction remains almost the same (Jain and Wallace, 2019; Wiegreffe and Pinter, 2019). Therefore, an optimal student unnecessarily shares similar attention matrices with its teacher. Our proposed AD-KD adopts a more reliable gradient-based method to obtain the attribution maps, which is shown better than attention matrices employed by baselines.

heading: C More Experimental Results
text: C.1 Results on MultiRC
Considering that the text in GLUE is relatively short (with Max_Seq_Length set to 128), We conduct additional experiments on SuperGLUE (Wang et al., 2019) for more comprehensive evaluation. We select a challenging QA task, MultiRC (Khashabi et al., 2018), with much longer text (with Max_Seq_Length set to 512) which requires more attribution knowledge. As shown in Table 6, ADKD improves 0.97% over vanilla KD and 0.38% over MGSKD. Moreover, the performance of ADKD is on par with the teacher.
C.2 Overfitting Study In this section, we investigate whether the overfitting problem would happen in attribution distillation. Using Eq. (11), we calculate the attribution gap between the teacher and the student models on the training and development sets of MRPC and QNLI respectively, and show the results in Figure 9. By altering β, the tendency of attribution gap on development sets is consistent with the one on training sets, which indicates that the attribution knowledge learned from training data can be well generalized to unseen data. Therefore, overfitting tends not to happen in attribution distillation.
C.3 Case Study In this section, we provide two examples to show how AD-KD facilitates the imitation of the teacher’s reasoning and outperforms vanilla KD. As shown in Figure 10, vanilla KD makes mistakes by ignoring keyword Louisiana or emphasizing an irrelevant word billion. In contrast, the attribution maps of AD-KD are more consistent with the ones in the teacher. AD-KD learns what to and not to focus on and thus predicts the label correctly.
ACL 2023 Responsible NLP Checklist

heading: A For every submission:

heading: 3 A1. Did you describe the limitations of your work?
text: Section Limitations

heading: 3 A2. Did you discuss any potential risks of your work?
text: Section Ethics Statement

heading: 3 A3. Do the abstract and introduction summarize the paper’s main claims?
text: Section Abstract and Section 1
7 A4. Have you used AI writing assistants when working on this paper? Left blank.
B 3 Did you use or create scientific artifacts? Section 4

heading: 3 B1. Did you cite the creators of artifacts you used?
text: Section 4.1 and Section 4.3
B2. Did you discuss the license or terms for use and / or distribution of any artifacts? Not applicable. Left blank.
7 B3. Did you discuss if your use of existing artifact(s) was consistent with their intended use, provided that it was specified? For the artifacts you create, do you specify intended use and whether that is compatible with the original access conditions (in particular, derivatives of data accessed for research purposes should not be used outside of research contexts)? The datasets we use are consistent with those used in previous works.
B4. Did you discuss the steps taken to check whether the data that was collected / used contains any information that names or uniquely identifies individual people or offensive content, and the steps taken to protect / anonymize it? Not applicable. Left blank.

heading: 3 B5. Did you provide documentation of the artifacts, e.g., coverage of domains, languages, and
text: linguistic phenomena, demographic groups represented, etc.? Section 4.1
3 B6. Did you report relevant statistics like the number of examples, details of train / test / dev splits, etc. for the data that you used / created? Even for commonly-used benchmark datasets, include the number of examples in train / validation / test splits, as these provide necessary context for a reader to understand experimental results. For example, small differences in accuracy on large test sets may be significant, while on small test sets they may not be. Appendix A.1
C 3 Did you run computational experiments? Section 5

heading: 3 C1. Did you report the number of parameters in the models used, the total computational budget
text: (e.g., GPU hours), and computing infrastructure used? Appendix A.2
The Responsible NLP Checklist used at ACL 2023 is adopted from NAACL 2022, with the addition of a question on AI writing assistance.

heading: 3 C2. Did you discuss the experimental setup, including hyperparameter search and best-found
text: hyperparameter values? Section 4.3 and Appendix A.2

heading: 3 C3. Did you report descriptive statistics about your results (e.g., error bars around results, summary
text: statistics from sets of experiments), and is it transparent whether you are reporting the max, mean, etc. or just a single run? Section 5.1

heading: 3 C4. If you used existing packages (e.g., for preprocessing, for normalization, or for evaluation), did
text: you report the implementation, model, and parameter settings used (e.g., NLTK, Spacy, ROUGE, etc.)? Section 4.3
D 7 Did you use human annotators (e.g., crowdworkers) or research with human participants? Left blank.
D1. Did you report the full text of instructions given to participants, including e.g., screenshots, disclaimers of any risks to participants or annotators, etc.? No response.
D2. Did you report information about how you recruited (e.g., crowdsourcing platform, students) and paid participants, and discuss if such payment is adequate given the participants’ demographic (e.g., country of residence)? No response.
D3. Did you discuss whether and how consent was obtained from people whose data you’re using/curating? For example, if you collected data via crowdsourcing, did your instructions to crowdworkers explain how the data would be used? No response.
D4. Was the data collection protocol approved (or determined exempt) by an ethics review board? No response.
D5. Did you report the basic demographic and geographic characteristics of the annotator population that is the source of the data? No response.

heading: Future Work
text: We will leave it to future work