Venue: NAACL 2021

# Multi-Grained Knowledge Distillation for Named Entity Recognition

Xuan Zhou \({}^{1}\), Xiao Zhang\({}^{1}\), Chenyang Tao\({}^{2}\), Junya Chen\({}^{2}\), Bing Xu\({}^{1}\), Wei Wang\({}^{1}\), and Jing Xiao\({}^{1}\)

###### Abstract

Although pre-trained big models (_e.g._, BERT, ERNIE, XLNet, GPT3 etc.) have delivered top performance in Seq2seq modeling, their deployments in real-world applications are often hindered by the excessive computations and memory demand involved. For many applications, including named entity recognition (NER), matching the state-of-the-art result under budget has attracted considerable attention. Drawing power from the recent advance in knowledge distillation (KD), this work presents a novel distillation scheme to efficiently transfer the knowledge learned from big models to their more affordable counterpart. Our solution highlights the construction of _surrogate labels_ through the \(k\)-best Viterbi algorithm to distill knowledge from the teacher model. To maximally assimilate knowledge into the student model, we propose a multi-grained distillation scheme, which integrates cross entropy involved in conditional random field (CRF) and fuzzy learning. To validate the effectiveness of our proposal, we conducted a comprehensive evaluation on five NER benchmarks, reporting cross-the-board performance gains relative to competing prior-arts. We further discuss ablation results to dissect our gains.

Ping An Technology (Shenzhen) Co., Ltd

2Duke University

{zhouxuan553,zhangxiao585,xubing962,wangwei192,xiaojting661}@pingan.com.cn

{chenyang.tao,junya.chen}@duke.edu

## 1 Introduction

The task of named entity recognition (NER) aims to put named entity mentioned in a sentence into some pre-defined categories such as the person names, organizations, locations, etc. NER is a fundamental task in nature language processing (NLP), and often serves as an information extraction tool embedded in solutions for downstream tasks such as event recognition and aspect-level sentiment analysis. And therefore, the computational and memory efficiency of an NER model is often considered crucial in many empirical settings.

Given its practical significance, considerable research effort has been devoted to NER in recent years. One fruitful direction is to boost NER performance through the use of more sophisticated model architectures, such as Transformer and its variants (_e.g._, BERT Devlin et al. (2019), ERNIE Sun et al. (2019c), XLNet Yang et al. (2019), RoBERTa Liu et al. (2019b), GPT3 Brown et al. (2020), etc.). Many of these models are based on the attention mechanism, which allows the models to adaptively focus on different parts of the sentence based on its current understanding. This enables more accurate parsing of the context, which is critical for the NER task. While such advanced models have delivered substantial improvements, a major drawback is that they are typically computationally expensive and memory intensive, preventing their applications in many cost-sensitive settings.

As such, it is often desirable to reduce those big models into more affordable counterparts, preferably without any significant performance drop. One strategy is to compress or truncate the original model, examples in this category include parameter pruning Srinivas and Babu (2015); McCarley (2019), low-rank approximation Yu et al. (2017); Ma et al. (2019) and parameter quantization Gong et al. (2014); Wu et al. (2016). The resulting model has a similar architecture to the original model, and consequently may suffer from similar limitations of the original model. Alternatively, knowledge distillation (KD) considers transferring knowledge into models with heterogeneous architectures Hinton et al. (2015), which allows more flexibility in the control of resource usage. In KD, a new learner, often dubbed the student model, assimilates knowledge from a pre-trained model, commonly known as the teacher. For classification tasks, this is typically achieved via minimizing their discrepancy in the output, regardless of the internal model architectures. Teacher outputs, sometimes referred to as the soft-labels, typically encode more information than what a student might receive from the

raw training feature-label pairs, thus resulting the improved learning efficiency.

However, for sequence tasks such as NER, standard techniques from KD do not readily apply. While common KD seeks to minimize the KL divergence between the output label distributions, for NER, the number of label combinations grows exponentially wrt the sequence length. Extracting teacher knowledge as if each combination is a different label category would be largely inefficient, if possible at all. On the other hand, many SOTA NER models are built on conditional random fields (CRF) to incorporate the label dependencies, and it only offers an un-normalized likelihood that is not directly amenable to the computation of KL-divergence. Another challenge for training an NER model is the lack of labeled sequences for training, while the unlabeled sequences may be plentiful. The promise to leverage unlabeled data to improve NER accuracy is appealing.

To address the above challenges, in this paper, we present an efficient knowledge distillation scheme that trains a light model (_e.g._, BiLSTM) which is able to retain the accuracy of its heavier counterparts, such as BERT, while significantly reduce cost. Our solution exploits the "soft surrogates", _i.e._, the most probable label sequences under the teacher model, to inform the student learner. To efficiently identify the most likely label sequences and determine their relative likelihood, we explore the use of Viterbi algorithm to expedite computation. We also explore the use of unlabeled data to improve performance.

In summary, we highlight the following contributions in this study: \((i)\) We present a novel multi-grained knowledge distillation strategy for sequence labeling via efficiently selecting \(k\)-best label sequence using Viterbi algorithm; \((ii)\) We advocate the use of a complete cross entropy loss and fuzzy distillation loss to respectively account for probability mass of un-selected sequences and the uncertainties in teacher confidence; \((iii)\) We present a comprehensive empirical analysis to dissect the gains from each individual components. We also show our model delivers substantial improvement relative to competing solutions on a wide-range of real-world benchmarks to demonstrate its utility.

## 2 Background

Our work is inspired by three lines of research: sequence-level knowledge distillation, \(k\)-best Viterbi algorithm and fuzzy conditional random field (CRF). In the following we review the technical backgrounds that are needed for the construction of our model.

**Sequence-level Knowledge Distillation.** Knowledge distillation (KD) is originally developed for classification tasks Hinton et al. (2015); Tang et al. (2019). When dealing with sequence outputs (_e.g._, machine translation, sequence labeling), where each unique combination of the output sequence is treated as different category, then the standard distillation objective is no longer appropriate, if feasible at all. This is because the number of unique combinations for a length \(L\) sequence with \(T\) possible tags scale at \(T^{L}\). To combat such exponential scaling, Kim and Rush (2016) investigated using beam search for teacher output to select \(k\)-best candidates for KD in neural machine translation (NMT), and Mun'im et al. (2019) utilized an similar technique for KD in Large Vocabulary Continuous Speech Recognition (LVSCR) tasks.

\(k\)**-best Viterbi Decoding.** Viterbi decoding is a dynamic programming technique to find a sequence with the highest score in an exponential-growth domain, with only linear complexity Viterbi (1967). Generalization has been proposed to extend its original scope to find the top-\(K\) sequences that are most probable, see Huang and Chiang (2005); Nielsen (2011) for details. A summary of the algorithm can be found in the supplementary material. In the context of KD in sequence tasks, a trained teacher model assigns varying probability to all sequence combinations. Our motivation is that the \(k\)-best Viterbi can be repurposed to pick out the \(K\)-most probable label sequences predicted by the teacher model, which plays an analogous role to the _softlabels_ (Figure 0(a)), without incurring unmanageable computational overhead.

**Conditional Random Field.** CRF Lafferty et al. (2001) is a classic and powerful energy-based model that is capable of capturing complex spatial or temporal dependency structures. In the context of NLP, it has been generally used as a refinement layer that accounts for correlations missed by the base NLP model, which typically brings in additional performance gains for various NLP tasks Lafferty et al. (2001); Collobert et al. (2011); Huang et al. (2015). Implementation-wise, CRF computes an energy given a candidate output \(\mathbf{y}\) and a context \(\mathbf{x}\) (_i.e._, input sequence), followed by a _softmax_ operator to obtain the conditional likelihood, _i.e._, 

\[{\mathbb{P}}(Y|{\bf x})=\frac{e^{s({\bf x},{\bf y})}}{\sum_{ \tilde{\mathbf{y}}\in Y_{\rm all}}e^{s({\bf x},\bar{\mathbf{y}})}}, \tag{1}\]

where \(Y_{\rm all}\) is the set of all possible tag sequences, \(s({\bf x},{\bf y})\) represents the _"compliance"_ energy score between two sequences \({\bf x}\) and \({\bf y}\). In classical NLP models, \(s({\bf x},{\bf y})\) is typically specified via handcrafted features and dependencies. In modern deep learning models, a typical decomposition of \(s({\bf x},{\bf y})\) is given by two parts

\[s({\bf x},{\bf y})=\sum_{j=1}^{L-1}\text{logit}_{j,{\bf y}_{j}}({ \bf x})+A_{{\bf y}_{j},{\bf y}_{j+1}}, \tag{2}\]

namely the _emission_ and _transition_ score. The transition matrix \(A\in{\mathbb{R}}^{T\times T}\) characterizes the smoothness of the label sequence (probability of switching between consequent labels), and \(\text{logit}_{j,k}({\bf x})\) denotes the likelihood of seeing label \(k\) at position \(j\) as predicted by the base model. Note that we do not have to enumerate the exponentially many \(Y_{\rm all}\) to compute the likelihood, which can be efficiently handled by the Viterbi algorithm introduced above in linear time Collobert et al. (2011b).

**Fuzzy CRF.** It has been argued that properly adjusting the uncertainty of teacher prediction usually lends better knowledge transfer to the student, which can either be sharpening or relaxing the teacher predicted distributions. In standard KD, this is achieved by the incorporation of an annealing factor. In our setup, we consider relaxation via generalizing CRF to a candidate set of label sequences Shang et al. (2018) rather than individual ones. Formally, we define the fuzzy loss as

\[{\mathbb{P}}(Y_{\rm candidate}|{\bf x})=\frac{\sum_{{\bf y}^{ \prime}\in Y_{\rm candidate}}e^{s({\bf x},{\bf y}^{\prime})}}{\sum_{\hat{\bm {y}}\in Y_{\rm all}}e^{s({\bf x},\bar{\mathbf{y}})}}, \tag{3}\]

where \(Y_{\rm candidate}\) contains candidate label sequences. Here, we will use the \(k\)-best teacher predicted label sequences as the candidate set.

## 3 Multi-Grained Distillation

In this section we detail the construction of our distillation scheme, with the overall architecture outlined in Figure 3.

### Viterbi Distillation for Sequence Outputs

Now we are ready to present details of our main contribution, Viterbi distillation for sequence outputs. Our basic idea is to extract information from the teacher model via drawing a set of most probable sequences, together with the respective confidence to those sequences. Then these sequences are presented to the student model during its training, to pass on the knowledge from teacher through various loss functions.

More specifically, we apply the \(k\)-best Viterbi algorithm (see Algorithm 1 in the supplementary material) to get the pairs \(\{({\mathbf{y}}_{i}^{(i)},p_{1}^{(i;t)}),\cdots,({\mathbf{y}}_{K}^{(i)},p_{ K}^{(i;t)})\}\) for sample \({\bf x}^{(i)}\), where \({\mathbf{y}}_{j}^{(i)}\) is the \(j\)-th most-likely label sequence, and \(p_{j}^{(i,{\bf t})}\) is the corresponding probability under the **teacher** model (indicated by the superscript \({\bf t}\)). Similarly, in our subsequent discussions we denote \(p_{j}^{(i,{\bf s})}\) as the probability produced by the **student** model on the label sequence \({\mathbf{y}}_{j}^{(i)}\).

#### 3.1.1 Fine-grained \(k\)-best Cross Entropy Distillation

We proposed to approximate the complete cross entropy with the following \(k\)-best Vertibi approximation to avoid the exponential level of computational

Figure 1: **Left. The model prediction accuracy improves as one includes more candidate \(K\). Right. The non-top-\(K\) probability remains non-neglectable even for moderately large \(K\). Here, OntoNotes 4.0 is used for illustration, and Y-axis represents the proportion of samples, whose non-top-\(K\) cumulative probability is greater than \(0.1\).**

Figure 2: \(k\)-best cross entropy loss uses predicted weights for each candidate label sequence, while \(k\)-best fuzzy loss group the candidates together and use their aggregated weight. The two losses represent fine-grained and coarse-grained learning from teacher respectively. Both schemes lump weights for the non-top-\(K\) labels.

 

complexity:

\[\begin{split}&\mathcal{L}_{\text{Seq-ce}}(\mathbf{x}^{(i)})\\ &:=-\sum_{\tilde{\boldsymbol{y}}\in Y_{\text{all}}}\mathbb{P}^{ \textbf{t}}(\tilde{\boldsymbol{y}}|\mathbf{x}^{(i)})\log\left(\mathbb{P}^{ \boldsymbol{s}}(\tilde{\boldsymbol{y}}|\mathbf{x}^{(i)})\right)\\ &\approx-\sum_{j=1}^{K}p_{j}^{(i,\textbf{t})}\log\left(p_{j}^{( i,\boldsymbol{s})}(\mathbf{y}_{j}^{(i)}|\mathbf{x}^{(i)})\right)\\ &\quad-\left(1-p_{\text{sum}}^{\textbf{t}}\right)\log\left(1-p_{ \text{sum}}^{\textbf{s}}\right),\end{split} \tag{4}\]

where \(\mathbb{P}^{\textbf{t}}(\cdot|\cdot)\) and \(\mathbb{P}^{\boldsymbol{s}}(\cdot|\cdot)\) are the CRF likelihoods respectively for teacher and student, and we collect the probability mass of the top-\(K\) sequences into terms \(p_{\text{sum}}^{\textbf{t}}\) and \(p_{\text{sum}}^{\textbf{s}}\), _i.e._,

\[p_{\text{sum}}^{\textbf{t}}=\sum_{j=1}^{K}p_{j}^{(i,\textbf{t})},\quad p_{ \text{sum}}^{\textbf{s}}=\sum_{j=1}^{K}p_{j}^{(i,\textbf{s})}. \tag{5}\]

Note that we have suppressed the dependency on **x** for notational clarity. We emphasize the residual probability mass \((1-p_{\text{sum}}^{\textbf{t}})\gg 0\) (see Figure 1b), and therefore should not be excluded from the loss as in Kim and Rush (2016); Wang et al. (2020).

#### 3.1.2 Coarse-grained Fuzzy Distillation

To more appropriately account for the uncertainty in the teacher guidance, we advocate the use of a fuzzy objective that does not discriminate between the likelihood among the top-\(K\) picks. Concretely, this is given by the binary cross entropy in terms of \(p_{\text{sum}}^{\textbf{t}}\) and \(p_{\text{sum}}^{\textbf{s}}\),

\[\begin{split}\mathcal{L}_{\text{Seq-fuzzy}}(\mathbf{x}^{(i)}):=-p _{\text{sum}}^{\textbf{t}}\log(p_{\text{sum}}^{\textbf{s}})\\ -\left(1-p_{\text{sum}}^{\textbf{t}}\right)\log\left(1-p_{\text{ sum}}^{\textbf{s}}\right).\end{split} \tag{6}\]

Multi-grained DistillationSee Figure 2 for a graphical illustration of cross entropy and fuzzy objectives, which shows different granularity on learning from teacher.

#### 3.1.3 Integrating Distillation Objectives

In addition to the "pure" distillation operations, we also allow direct learning from ground-truth labels and surrogate ones labeled by the teacher model (details found in Section 3.2). These two kinds of targets are named hard labels, and the related loss is as follows,

\[\mathcal{L}_{\text{hard}}(\mathbf{x}^{(i)}):=-\log\mathbb{P}(\mathbf{y}_{ \text{hard}}|\mathbf{x}^{(i)}). \tag{7}\]

Thus, the final loss is the weighted sum of three terms defined above,

\[\begin{split}&\mathcal{L}_{\text{Seq-all}}=\frac{1}{N}\sum_{i=1}^ {N}\lambda_{1}\mathcal{L}_{\text{hard}}(\mathbf{x}^{(i)})\\ +&\lambda_{2}\mathcal{L}_{\text{Seq-fuzzy}}(\mathbf{ x}^{(i)})+\lambda_{3}\mathcal{L}_{\text{Seq-cge}}(\mathbf{x}^{(i)}),\end{split} \tag{8}\]

where \(\lambda_{i}\geq 0,i=1,2,3\).

Automated Tuning of Loss WeightsTuning the loss weights \(\{\lambda_{i}\}_{i=1}^{3}\) in (8) via standard techniques such as grid search can be laborious and costly. In this work, we leverage the _uncertainty weighting_ strategy proposed in Cipolla et al. (2018) to automate the weight selection procedure to balance multi-task objectives, both for its simplicity and robust performance. More specifically, we add uncertainty regularization terms for the weights, resulting

\[\begin{split}\mathcal{L}_{\text{Seq-all}}^{\text{u}}=\mathcal{ L}_{\text{Seq-all}}-\frac{1}{2}\left(\log\lambda_{1}+\log\lambda_{2}+\log \lambda_{3}\right),\end{split} \tag{9}\]

as our new learning objective.

### Data Augmentation and Misc

Augmenting with Unlabeled DataNER applications are often challenged with the lack of labeled instances for training. Motivated by the observations from prior studies that data augmentation using unlabeled data may improve distillation performance Mukherjee and Awadallah (2019); Tang

Figure 3: Model schematic for multi-grained knowledge distillation.

 

et al., 2019), we also feed unlabeled data to the teacher model to construct additional surrogate label sequences using \(k\)-best Viterbi for distillation. We pool the surrogate sequences from both labeled and unlabeled data for cross entropy and fuzzy loss. Besides, we use Top-\(1\) sequence as hard label to educate the student.

**Token-level Distillation.** To verify the necessity of using a sequence-level distillation, we additionally consider token-level distillation with standard KD techniques. Specifically, we consider the following loss

\[{\mathcal{L}_{\text{Token}}:=}-\frac{1}{N}\sum_{i=1}^{N}\sum_{j=1}^{T}{ \mathcal{L}_{i,j}}, \tag{10}\]

where \({\mathcal{L}_{i,j}=\sum_{k=1}^{L}\mathbb{P}^{4}(\mathbf{y}_{j}=k|\mathbf{x}^ {(i)})\log(\mathbb{P}^{8}(\mathbf{y}_{j}=k|\mathbf{x}^{(i)}))}\) is the token-level KL-divergence between the teacher's and student's predicted distributions for each token, rather than the token sequence distribution. This breaks the dependencies between tokens to avoid the combinatory complexity, thus enabling standard KD. The probability \(\mathbb{P}(\mathbf{y}_{j}=k|\mathbf{x}^{(i)})\) can be simply calculated by _emission_ score, in this case, the loss is denoted as \({\mathcal{L}_{\text{Token-em}}}\). To recover the spatial dependencies, we can add _transition_ matrix to derive the probability \(\mathbb{P}(\mathbf{y}_{j}=k|\mathbf{x}^{(i)})\) by calculating the marginal distribution of token from the probability distribution \(\mathbb{P}(\mathbf{y}|\mathbf{x}^{(i)})\) given by CRF, similar to Wang et al. (2020). We denote this corresponding loss as \({\mathcal{L}_{\text{Token-pos}}}\).

## 4 Related Work

**NER without BERT.** Extensive investigations in NER have been conducted without appealing to the BERT architecture, and instead using simpler architectures such as BiLSTM or convolutional neural network (CNN) Huang et al. (2015); Strubell et al. (2017a). Special effort has been made to adapt the NER network architectures for better handling of lexicons. Liu et al. (2019) leveraged hybrid semi-Markov CRFs to improve NER recognition with Gazetteers, where segments are used as the basic units instead of words. In Zhang and Yang (2018) a lattice-structured LSTM is proposed to encode a sequence of input characters as well as all potential words that match a lexicon. Gui et al. (2019) incorporates lexicons using a rethinking mechanism under a CNN setup, which renders faster inference compared with lattice-LSTM. Peng et al. (2019) explores weighted word embedding to match all probable words given the lexicons. In the method given by Ghaddar and Langlais (2018), a lexical representation is computed for each word with a 120-dimensional vector, where element encodes the similarity of the word with an entity type. Recently, Ding et al. (2019) leveraged the graph neural network (GNN) to exploit the additional rich information captured by the gazetteers, setting new SOTA performance. Our distillation work is orthogonal to these developments, and the techniques can be combined for further improvements.

**Compressing BERT with Distillation.** Various efforts have been made to reduce the size and cost of BERT or other big models while maximally maintaining their outstanding performance, and KD offers an appealing alternative to the direct amputation of the original models. Along this line, Tang et al. (2019) studied on compressing BERT to BiLSTM for classification, resulting a model with comparable performance to ELMo but \(100\times\) fewer parameters and \(15\times\) faster inference. Tsai et al. (2019) successfully applied KD for multilingual sequence labeling model, enabling SOTA results on an MiniBERT model afforded by a single CPU. Other developments include distillation on intermediate representations Sun et al. (2019) and student pre-training Turc et al. (2019). The value of unlabeled data in KD has also been explored Mukherjee and Awadallah (2019). Our work complements these studies via presenting a framework of using top-\(K\) soft-surrogate labels for KD.

**Distilling with top-\(K\) Picks.** Parallel to our work, Tang et al. (2020) also considered a top-\(K\) scheme for KD. This work distincts from our proposal via assigning uniform weights to the un-selected labels sequences, a practice that can be largely inefficient. Another work close to our setup is Wang et al. (2020), where the authors tried to distill the structural knowledge from multiple monolingual teachers to a single student. In their loss, the probability mass of the non-top-\(K\) picks is discarded. In Figure 1 (right), we show that the probability for non-top-\(K\) picks is non-neglectable, which is properly accounted for in our proposed complete cross entropy loss.

## 5 Experiment

To validate the proposed solution and elaborate the gains, we benchmark it against state-of-the-art methods, through a wide range of experiments on 

real-world datasets. All experiments are implemented with Tensorflow and executed on a single NVIDIA P100 GPU. Details of the experimental setup are provided in supplementary material, due to space limits, and our code will be available at [https://github.com/llzhouxuan/multi_grained_kd_ner](https://github.com/llzhouxuan/multi_grained_kd_ner).

### Datasets

The following real-world datasets are considered in our study. Detailed summary statistics of the datasets can be found in supplementary material.

**CoNLL-2003 NER**(Sang and De Meulder, 2003) consists of newswire from Reuters RCV corpus. Unlabeled data from the Reuters RCV corpus is used for our data augmentation experiments.

**OntoNotes**(Consortium et al., 2011) is an annotated multilingual corpus consists of texts from a wide variety of sources, such as telephone conversation, broadcast and newswire. For our NER experiment, we consider

* **English NER** is derived from OntoNotes Release 5.0 and processed according to Pradhan et al. (2013).
* **Chinese NER** is derived from OntoNotes Release 4.0 and processed according to Che et al. (2013).

Text data from other OntoNotes tasks are used as unlabeled data.

**MSRA**(Levow, 2006) is a Chinese NER dataset with its corpus derived from news domain. We use the Chinese word segmentation dataset MSR Levow, 2006) for unlabeled data.

**Weibo**(Peng and Dredze, 2015) is a Chinese NER dataset derived from social media contents. Only a small fraction of data is labeled in this dataset.

### Model specification

We briefly describe the modeling and training specifications choices for the teacher and student models below.

Teacher Model.To build a strong learner, our teacher model is constructed by a BERT model followed by a CRF layer, and we denote it as BERT+CRF. Specifically, we use BERTBASE1 model as our feature encoder, which is known to perform strongly across a wide range of NLP tasks. A dropout layer is concatenated to the BERT, followed by a fully connected layer that computes the logit\(({\bf x})\) for the labels at each location. We further apply an additional CRF-layer as defined in (2) to account for the temporal dependencies among the labels, similar to the work of Meng et al. (2019). The teacher model is trained using the standardized fine-tune paradigm Devlin et al. (2019). Following Howard and Ruder (2018); Sun et al. (2019a), we set different learning rates for each layer. A larger rate is used for CRF, and for the BERT the learning rates decays by a factor of \(0.9\) as the layers approaches the input.

Footnote 1: [https://github.com/google-research/bert](https://github.com/google-research/bert)

Student Model.For our student model, we want it to be light, fast yet still sufficiently expressive. To this end, we use the BiLSTM+CRF architecture proposed in Huang et al. (2015). This model exploits a Bidirectional LSTM to map input sequence \({\bf x}\) into a sequence of feature vectors, which accounts for the context from both directions. The rest of the construction follows what has been described for the teacher model, with the BERT part replaced by the BiLSTM. We reuse the learned word-embeddings from the teacher model and keep it frozen during training. Empirically, we find this strategy produces better results, possibly due to reduced effort transferring the knowledge of richer embedding representation compared to alternatives such as _word2vec_, and it also avoids over-fitting.

### Baselines, Variations and Evaluation

In addition to the vanilla teacher and student models described above, we also considered the following strong established NER baselines in our experiments.

* **BERT teacher baselines (Chinese NER)** BERT+Glyce Meng et al. (2019).
* **Non-BERT student baselines (English NER)** LSTM-CNNs Chiu and Nichols (2016), LEX Ghaddar and Langlais (2018), IDCNN Strubell et al. (2017b), HSCF Liu et al. (2019).
* **Non-BERT student baselines (Chinese NER)** SUL Peng et al. (2019), Lattice-LSTM Zhang and Yang (2018), NMDM Ding et al. (2019), ME-CNER Xu et al. (2019).

Variations.To further understand the contributions from different components of our proposal, we run different variations of the model to dissect the gains. We use BiLSTM+CRF as our 

base model, and consider variants with combinations of the following components in our experiments: (\(i\)) TE: use fixed pre-trained teacher embedding; (\(ii\)) TOKEN-EM: token-level distillation with loss \({\cal L}_{\text{Token-em}}\); (\(iii\)) TOKEN-POS: token-level distillation with with loss \({\cal L}_{\text{Token-pos}}\); (\(iv\)) SEQ: multi-grained sequence-level distillation with loss \({\cal L}^{\rm u}_{\text{Seq-all}}\); (\(v\)) DA: augmented with unlabeled data. We also vary the size of best candidate set from \(K=1\) to \(K=15\).

Evaluation.We report the F1 score following Sang and De Meulder (2003), and relegate other quantitative metrics such as _Precision_ and _Recall_ to the supplementary material. We apply early stopping with max patience set to \(5\) based on the performance of the development set.

### Analysis of Results

From Table 1 we first find out that, in all three Chinese datasets, our teacher model outperforms two baselines Meng et al. (2019). We owe this to the layer-wise learning rate and discriminative fine-tuning strategies Howard and Ruder (2018); Sun et al. (2019a). Another analysis in terms of teacher is that directly copying the teacher embedding to the student model can be most helpful, for both English and Chinese datasets.

Regarding distillation, it achieved cross-the-board performance gains relative to the no-distillation TE baseline. Our final proposal, namely TE+DA+SEQ, performs better or similarly to almost all of its non-BERT baselines. Note that these competing baselines have leveraged additional domain knowledge, such as lexicon information, to boost NER performance (see Related Work). Such practice not only adds specialized modeling effort and complexity, but also make the resulting architecture less generalizable to other tasks. Our results show that we can match the performance via reaping the knowledge from more sophisticated models using general purpose sequence-level distillation, rather than appealing to dedicated modeling effort.

We also observed that inducing data augmentation consistently improves student learning. And notably, in all cases, the sequence-level distillation outperforms token-level distillation, especially in the absence of data augmentation.

\begin{table}
\begin{tabular}{l l|c c|c c c} \hline \multicolumn{4}{c|}{**Models**} & \multicolumn{6}{c}{**Datasets**} \\ \cline{3-7}  & & \multicolumn{2}{c|}{**English Datasets**} & \multicolumn{3}{c}{**Chinese Datasets**} \\ \hline \multirow{2}{*}{**English**} & **Chinese** & **CoNLL-2003** & **OntoNotes 5.0** & **MSRA** & **Weibo** & **OntoNotes 4.0** \\ \hline \multirow{2}{*}{BERT+CRF (teacher)} & Vanilla BERT+CRF & NA & NA & 94.80 & 67.33 & 79.16 \\  & Glyce+BERT & NA & NA & 95.54 & 67.60 & 80.62 \\  & BERT+CRF (teacher) & **92.03** & **89.92** & **95.91** & **71.22** & **82.39** \\ \hline LSTM-CNNs & Lattice-LSTM & 90.91 & 86.17 & 93.18 & 58.79 & 73.88 \\ IDCNN & NMDM & 90.54 & 86.84 & **94.4** & 59.5 & **76.0** \\ LEX & SUL & 90.52 & 87.95 & 93.44 & 61.24 & 75.54 \\ HSCRF & ME-CNER & **91.10** & **89.94** & 91.45 & **68.93** & NA \\ \hline Vanila BLSTM+CRF & 80.08 & 82.58 & 90.70 & 51.85 & 64.40 \\ + TE (no distilation) & 88.35 \(\uparrow\) 8.27 & 87.35 \(\downarrow\) 4.77 & 91.69 \(\uparrow\) 0.99 & 54.87 \(\uparrow\) 3.02 & 69.78 \(\uparrow\) 5.38 \\ + TE + TOKEN-EM & 88.46 \(\uparrow\) 8.38 & 87.39 \(\uparrow\) 4.81 & 91.51 \(\uparrow\) 0.81 & 53.55 \(\uparrow\) 1.70 & 70.46 \(\uparrow\) 6.06 \\ + TE + TOKEN-POS & 88.66 \(\uparrow\) 8.58 & 88.17 \(\uparrow\) 5.59 & 91.77 \(\uparrow\) 1.07 & 54.34 \(\uparrow\) 2.49 & 71.19 \(\uparrow\) 6.79 \\ + TE + SEQ & **89.54** \(\uparrow\) 9.46 & **88.34** \(\uparrow\) 5.76 & **91.98** \(\uparrow\) 1.98 & **57.14** \(\uparrow\) 5.29 & **72.33** \(\uparrow\) 7.93 \\ + TE + DA (no distilation) & 90.69 \(\uparrow\) 10.61 & 87.52 \(\downarrow\) 4.94 & 92.68 \(\uparrow\) 1.96 & 69.74 \(\uparrow\) 17.89 & 74.53 \(\uparrow\) 1.03 \\ + TE + DA + TOKEN-EM & 90.49 \(\uparrow\) 10.41 & 88.05 \(\uparrow\) 5.47 & 92.66 \(\uparrow\) 1.96 & 69.83 \(\uparrow\) 17.98 & 74.52 \(\uparrow\) 10.12 \\ + TE + DA + TOKEN-POS & 90.98 \(\uparrow\) 10.90 & 88.29 \(\uparrow\) 5.71 & 92.59 \(\uparrow\) 1.89 & 69.90 \(\uparrow\) 18.05 & 74.88 \(\uparrow\) 10.48 \\ + TE + DA + SEQ (student) & **91.17** \(\uparrow\) 11.09 & **88.91** \(\uparrow\) 6.32 & **92.99** \(\uparrow\) 2.29 & **71.62** \(\uparrow\) 19.77 & **76.05** \(\uparrow\) 11.65 \\ \hline \end{tabular}
\end{table}
Table 1: Comparison of F1 scores. \(\uparrow\) denotes the gain relative to Vanilla BiLSTM+CRF baseline. Best results in each category are shown in bold. Results for the competing baselines are collected from the original papers.

\begin{table}
\begin{tabular}{l|c c c|c c} \hline \multirow{2}{*}{Loss} & \multicolumn{4}{c}{Datasets} \\ \cline{2-5}  & \multicolumn{2}{c|}{OntoNotes 4.0} & \multicolumn{2}{c}{CoNLL-2003} \\ \cline{2-5}  & P & R & FI & P & R & FI \\ \hline \(L^{\rm a}_{\text{Sq-ball}}\) & 77.96 & 74.23 & **76.05** & **90.85** & 91.50 & **91.17** \\   + \({\cal L}_{\text{Token-pos}}\) & 76.63 & 74.47 & 75.53 & 90.63 & 91.51 & 91.07 \\   - \({\cal L}_{\text{Seq-frany}}\) & **78.11** & 73.43 & 75.70 & 90.43 & 91.11 & 90.77 \\   - \({\cal L}_{\text{Seq-oc}}\) & 75.71 & **74.87** & 75.29 & 90.35 & **91.60** & 90.97 \\ \hline \end{tabular}
\end{table}
Table 2: Performance with different loss combinations.

\begin{table}
\begin{tabular}{l|c c|c c} \hline \multirow{2}{*}{Datasets} & \multicolumn{2}{c|}{Without DA} & \multicolumn{2}{c}{With DA} \\ \cline{2-5}  & auto & equal & auto & equal \\ \hline MSRA & 91.98 & 91.77 & 92.99 & 92.83 \\ Weibo & 57.14 & 55.59 & 71.62 & 70.09 \\ OntoNotes 4.0 & 72.32 & 71.32 & 76.05 & 75.83 \\ CoNLL-2003 & 89.54 & 89.26 & 91.17 & 91.09 \\ OntoNote 5.0 & 88.34 & 88.27 & 88.91 & 88.42 \\ \hline \end{tabular}
\end{table}
Table 3: Comparison of automated uncertainty weighting (auto) defined in (9) versus equal weighting (equal).

 



### Ablation Study

**Size of Candidate Set.** In Figure 4, we compare how varying the size of candidate set affects performance. The performance peaks at a moderate \(K\), after that the gain tapers off. This is because when using an excessive \(K\), the distillation process starts to include more inaccurate label-sequences, compromising the learning efficiency (as top-\(K\) label-sequences are treated equally in the fuzzy objective).

**Effect of Augmenting with Unlabeled Data.** In Figure 5, we report the F1 scores with different number of unlabeled instances to augment the distillation phase for on Weibo dateset, which has very few training instances but an enormous unlabeled set. The use of data augmentation drastically boosts student's performance (F1: \(57.14\to 71.62\)). The distillation performance monotonically increases as more unlabeled instances are used, and the performance closes, even beats the teacher model in the large sample limit.

**Loss Combinations.** We further explore additional combinations of losses to sharpen our understanding, with main results summarized in Table 2. Combinations of sequence-level and token-level loss (_i.e._, \({\mathcal{L}^{\text{u}}_{\text{Seq-all}}+{\mathcal{L}_{\text{Token-pos}}}}\)) reveal a slight drop in performance. We also subtract each individual \({\mathcal{L}_{\text{Seq-fuzzy}}}\) and \({\mathcal{L}_{\text{Seq-ce}}}\) from \({\mathcal{L}^{\text{u}}_{\text{Seq-all}}}\). It appears that the coarse-grained loss \({\mathcal{L}_{\text{Seq-fuzzy}}}\) is propitious to increase the recall value, while using the fine-grained loss \({\mathcal{L}_{\text{Seq-ce}}}\) will get better precision value.

**Effect of Uncertainty Weighting.** In this paper, we balance our loss functions with different scales by injecting uncertainty Cipolla et al. (2018). This method yields a regularization term of weight, preventing the weight tends to zero. Table 3 shows that, regularized weighting outperforms equal weighting in all cases.

**Computation Efficiency.** To examine the efficiency gains, we compare inference time difference between the teacher and student model, as reported in Table 4. Our student model achieves over \(40\times\) reduction and speedup, while achieving comparable performance compared with the teacher model.

**Alternative Choice of Student Model.** We also exam the performance of multi-grained distillation method on another student model, that is four transformer layers, which has a similarity structure as BERT. The results are showed in supplementary material (Table 9).

## 6 Conclusions

In this work, we develop novel multi-grained knowledge distillation techniques to train a light NER model with comparable performance with their more sophisticated counterparts. In particular, we show that Viterbi algorithm can be exploited to impart the knowledge of \(k\)-best predictions from the teacher model to the student. We further advocated the use of CRF adjustments, fuzzy objective and data augmentation to improve performance. Our empirical experiments carefully analyze the gains from our proposal on a wide range of NER benchmarks, via efficiently transferring knowledge from a powerful BERT model to a much more compact BiLSTM student. In future work, we seek to extend the proposed framework to more general distillation applications where CRF is used, such as speech recognition, and distill with more general representation transfer schemes Chen et al. (2020).

Figure 4: Performance with different candidate size \(K\).

Figure 5: Performance gains using more unlabeled data on the Weibo dataset. Dashed line denotes the teacher performance baseline.





## Acknowledgements

The authors would like to thank the anonymous reviewers for their insightful comments. The research conducted at Duke was supported in part by the DOE, NSF and ONR.

## References

* Brown et al. (2020) Tom B Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. 2020. Language models are few-shot learners. _arXiv preprint arXiv:2005.14165_.
* Che et al. (2013) Wanxiang Che, Mengqiu Wang, Christopher D Manning, and Ting Liu. 2013. Named entity recognition with bilingual constraints. In _NAACL_.
* Chen et al. (2020) Junya Chen, Zidi Xiu, Benjamin Goldstein, Ricardo Henao, Lawrence Carin, and Chenyang Tao. 2020. Supercharging imbalanced data learning with causal representation transfer. _arXiv preprint arXiv:2011.12454_.
* Chiu and Nichols (2016) Jason P.C. Chiu and Eric Nichols. 2016. Named entity recognition with bidirectional LSTM-CNNs. _Transactions of the Association for Computational Linguistics_, 4:357-370.
* Cipolla et al. (2018) R. Cipolla, Y. Gal, and A. Kendall. 2018. Multi-task learning using uncertainty to weigh losses for scene geometry and semantics. In _CVPR_.
* Collobert et al. (2011a) Ronan Collobert, Jason Weston, Leon Bottou, Michael Karlen, Koray Kavukcuoglu, and Pavel Kuksa. 2011a. Natural language processing (almost) from scratch. _Journal of machine learning research_, 12(ARTICLE):2493-2537.
* Collobert et al. (2011b) Ronan Collobert, Jason Weston, Leon Bottou, Michael Karlen, Koray Kavukcuoglu, and Pavel Kuksa. 2011b. Natural language processing (almost) from scratch. _Journal of machine learning research_, 12(Aug):2493-2537.
* Consortium et al. (2011) Linguistic Data Consortium et al. 2011. Ontonotes release 4.0 [online] available from: https://catalog. ldc.upenn. edu. _LDC2011T03_.
* Devlin et al. (2019) Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language understanding. In _NAACL_.
* Ding et al. (2019) Ruixue Ding, Pengjun Xie, Xiaoyan Zhang, Wei Lu, Linlin Li, and Luo Si. 2019. A neural multi-digraph model for chinese mer with gazetteers. In _ACL_.
* Ghaddar and Langlais (2018) Abbas Ghaddar and Phillippe Langlais. 2018. Robust lexical features for improved neural network named-entity recognition. In _COLING_.
* Gong et al. (2014) Yunchao Gong, Liu Liu, Ming Yang, and Lubomir Bourdev. 2014. Compressing deep convolutional networks using vector quantization. _arXiv preprint arXiv:1412.6115_.
* Gui et al. (2019) Tao Gui, Ruotian Ma, Qi Zhang, Lujun Zhao, Yu-Gang Jiang, and Xuanjing Huang. 2019. Cnn-based chinese ner with lexicon rethinking. In _IJCAI_.
* Hinton et al. (2015) Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. 2015. Distilling the knowledge in a neural network. _Computer Science_, 14(7):38-39.
* Howard and Ruder (2018) Jeremy Howard and Sebastian Ruder. 2018. Universal language model fine-tuning for text classification. In _ACL_.
* Huang and Chiang (2005) Liang Huang and David Chiang. 2005. Better k-best parsing. In _Proceedings of the Ninth International Workshop on Parsing Technology_.
* Huang et al. (2012) Zhiheng Huang, Yi Chang, Bo Long, Jean-Francois Crespo, Anlei Dong, Sathiya Keerthi, and Su-Lin Wu. 2012. Iterative Viterbi A* algorithm for k-best sequential decoding. In _ACL_.
* Huang et al. (2015) Zhiheng Huang, Wei Xu, and Kai Yu. 2015. Bidirectional LSTM-CRF models for sequence tagging. _CoRR_, abs/1508.01991.
* Jiao et al. (2019) Xiaoqi Jiao, Yichun Yin, Lifeng Shang, Xin Jiang, Xiao Chen, Linlin Li, Fang Wang, and Qun Liu. 2019. Tinybert: Distilling bert for natural language understanding. _arXiv preprint arXiv:1909.10351_.
* Kim and Rush (2016) Yoon Kim and Alexander M. Rush. 2016. Sequence-level knowledge distillation. In _EMNLP_.
* Lafferty et al. (2001) John Lafferty, Andrew McCallum, and Fernando Pereira. 2001. Conditional random fields: Probabilistic models for segmenting and labeling sequence data. In _ICML_.
* Levow (2006) Gina-Anne Levow. 2006. The third international Chinese language processing bakeoff: Word segmentation and named entity recognition. In _Proceedings of the Fifth SIGHAN Workshop on Chinese Language Processing_.
* Liu et al. (2019a) Tianyu Liu, Jin-Ge Yao, and Chin-Yew Lin. 2019a. Towards improving neural named entity recognition with gazetteers. In _ACL_.
* Liu et al. (2019b) Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019b. Roberta: A robustly optimized bert pretraining approach. _arXiv preprint arXiv:1907.11692_.
* Ma et al. (2019) Xingtian Ma, Peng Zhang, Shuai Zhang, Nan Duan, Yuexian Hou, Dawei Song, and Ming Zhou. 2019. A tensorized transformer for language modeling. _arXiv preprint arXiv:1906.09777_.
* McCarley (2019) JS McCarley. 2019. Pruning a bert-based question answering model. _arXiv preprint arXiv:1910.06360_.
* McCarley et al. (2019)

Yuxian Meng, Wei Wu, Fei Wang, Xiaoya Li, Ping Nie, Fan Yin, Muyu Li, Qinghong Han, Xiaofei Sun, and Jiwei Li. 2019. Glyce: Glyph-vectors for chinese character representations. In _NIPS_.
* Mukherjee and Awadallah (2019) Subhabrata Mukherjee and Ahmed Hassan Awadallah. 2019. Distilling transformers into simple neural networks with unlabeled transfer data. _arXiv preprint arXiv:1910.01769_.
* Mun'im et al. (2019) Raden Mu'az Mun'im, Nakamasa Inoue, and Koichi Shinoda. 2019. Sequence-level knowledge distillation for model compression of attention-based sequence-to-sequence speech recognition. In _ICASSP_.
* Nielsen (2011) Jesper Nielsen. 2011. A coarse-to-fine approach to computing the k-best viterbi paths. In _CPM_.
* Peng et al. (2019) Minlong Peng, Ruotian Ma, Qi Zhang, and Xuanjing Huang. 2019. Simplify the usage of lexicon in chinese mer. _arXiv preprint arXiv:1908.05969_.
* Peng and Dredze (2015) Nanyun Peng and Mark Dredze. 2015. Named entity recognition for chinese social media with jointly trained embeddings. In _EMNLP_.
* Pradhan et al. (2013) Sameer Pradhan, Alessandro Moschitti, Nianwen Xue, Hwee Tou Ng, Anders Bjorkelund, Olga Uryupina, Yuchen Zhang, and Zhi Zhong. 2013. Towards robust linguistic analysis using ontonotes. In _CoNLL_.
* Sang and De Meulder (2003) Erik F Sang and Fien De Meulder. 2003. Introduction to the conll-2003 shared task: Language-independent named entity recognition. _arXiv preprint cs/0306050_.
* Shang et al. (2018) Jingbo Shang, Liyuan Liu, Xiaotao Gu, Xiang Ren, Teng Ren, and Jiawei Han. 2018. Learning named entity tagger using domain-specific dictionary. In _EMNLP_.
* Srinivas and Babu (2015) Suraj Srinivas and R. Venkatesh Babu. 2015. Data-free parameter pruning for deep neural networks. _Computer Science_, pages 2830-2838.
* Strubell et al. (2017a) Emma Strubell, Patrick Verga, David Belanger, and Andrew McCallum. 2017a. Fast and accurate entity recognition with iterated dilated convolutions. In _EMNLP_.
* Strubell et al. (2017b) Emma Strubell, Patrick Verga, David Belanger, and Andrew McCallum. 2017b. Fast and accurate entity recognition with iterated dilated convolutions. In _EMNLP_.
* Sun et al. (2019a) Chi Sun, Xipeng Qiu, Yige Xu, and Xuanjing Huang. 2019a. How to fine-tune bert for text classification? _arXiv preprint arXiv:1905.05583_.
* Sun et al. (2019b) Siqi Sun, Yu Cheng, Zhe Gan, and Jingjing Liu. 2019b. Patient knowledge distillation for bert model compression. _arXiv preprint arXiv:1908.09355_.
* Sun et al. (2019c) Yu Sun, Shuohuan Wang, Yukun Li, Shikun Feng, Xuyi Chen, Han Zhang, Xin Tian, Danxiang Zhu, Hao Tian, and Hua Wu. 2019c. Ernie: Enhanced representation through knowledge integration. _arXiv preprint arXiv:1904.09223_.
* Tang et al. (2020) Jiaxi Tang, Rakesh Shivanna, Zhe Zhao, Dong Lin, Anima Singh, Ed H. Chi, and Sagar Jain. 2020. Understanding and Improving Knowledge Distillation. _arXiv:2002.03532 [cs, stat]_. ArXiv: 2002.03532.
* Tang et al. (2019) Raphael Tang, Yao Lu, Linqing Liu, Lili Mou, Olga Vechtomova, and Jimmy Lin. 2019. Distilling taskspecific knowledge from bert into simple neural networks. _arXiv preprint arXiv:1903.12136_.
* Tsai et al. (2019) Henry Tsai, Jason Riesa, Melvin Johnson, Naveeen Arivazhagan, Xin Li, and Amelia Archer. 2019. Small and Practical BERT Models for Sequence Labeling. In _EMNLP-IJCNLP_.
* Turc et al. (2019) Iulia Turc, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. Well-read students learn better: On the importance of pre-training compact models. _arXiv preprint arXiv:1908.08962_.
* Viterbi (1967) Andrew Viterbi. 1967. Error bounds for convolutional codes and an asymptotically optimum decoding algorithm. _IEEE transactions on Information Theory_, 13(2):260-269.
* Wang et al. (2020)Xinyu Wang, Yong Jiang, Nguyen Bach, Tao Wang, Fei Huang, and Kewei Tu. 2020. Structure-level knowledge distillation for multilingual sequence labeling. In _ACL_.
* Wu et al. (2016)Jiaxiang Wu, Cong Leng, Yuhang Wang, Qinghao Hu, and Jian Cheng. 2016. Quantized convolutional neural networks for mobile devices. In _CVPR_.
* Xu et al. (2019)Canwen Xu, Feiyang Wang, Jialong Han, and Chelliang Li. 2019. Exploiting multiple embeddings for chinese named entity recognition. In _CIKM_.
* Yang and Zhang (2018) Jie Yang and Yue Zhang. 2018. NCRF++: An open-source neural sequence labeling toolkit. In _ACL_.
* Yang et al. (2019) Zhilin Yang, Zihang Dai, Yaming Yang, Jaime Carbonell, Ruslan Salakhutdinov, and Quoc V Le. 2019. XLnet: Generalized autoregressive pretraining for language understanding. _arXiv preprint arXiv:1906.08237_.
* Yu et al. (2017)Xiyu Yu, Tongliang Liu, Xinchao Wang, and Dacheng Tao. 2017. On compressing deep models by low rank and sparse decomposition. In _CVPR_.
* Zhang and Yang (2018) Yue Zhang and Jie Yang. 2018. Chinese NER using lattice LSTM. In _ACL_.

## Appendix A Datasets statistic

Detailed summary statistics of the datasets 



## Appendix B Hyperparameters

In Table 6 we list the search grids for our hyperparameter tuning. For each dataset we evaluate all combinations and report the test set results for best-performing model on the validation set.

## Appendix C \(k\)-best Viterbi Algorithm

We implemented the \(k\)-best Viterbi algorithm generalized from the classic Viterbi algorithm, that is to storage the Top-K rather than maximum scores at each time step and tag type. Though exists more efficient \(k\)-best Viterbi implementation Huang et al. (2012) for large label scenarios, it only offers marginal efficiency gains as in our tasks. Unlike existing work Yang and Zhang (2018), we remove short examples where \(K\) is larger than the possible number of label sequences. In our application, we also output the probability rather than the path score, and this needs another dynamic programming to calculate the denominator in (1). The transition matrix is rule-constrained such that invalid transitions (_e.g._, B-ORG\(\rightarrow\) I-PER) are prohibited.

```
Input:The length of input sequence \({\bf x}^{(i)}\) as \(L\), the number of tags \(T\), transition matrix \(A[T\times T]\), the \(logits[L\times T]\) of \({\bf x}^{(i)}\), \(K\) Output:\(k\)-best paths and corresponding probabilities Initialize:CurrentPath=\(1:TG[0;z]\);
#\(m=0\)to\(L-1\)do
#\(w\)=CurrentPath.shape[1]; Update CurrentPath by adding \(T\) possible tags on the end of every path; Update CurrentScore of each path through (2); ifCurrentPath.shape[0] \(<K\times T\)then Continue
#end Group sort\(k\)-best on CurrentScore; Update CurrentPath using the indexes of \(k\)-best scores, resulting \([w+1,K\times T]\); Update CurrentScore using the indexes of \(k\)-best scores, resulting \([K,1]\);
#end Calculate the denominator in (1) using another dynamic programming, for details see Collobert et al. (2011b); return\(k\)-best paths and k-best probabilities or \(\{({\bf y}_{1}^{(i)},p_{1}^{(i,t)}),\cdots,({\bf y}_{K}^{(i)},p_{K}^{(i,t)})\}\),
```

**Algorithm 1**Pseudocode of \(k\)-best Viterbi Algorithm

## Appendix E Results on Sentiment Analysis Task

Table 7 shows the experimental results on dataset SemEval-2016 Task 5 of aspect based sentiment analysis. In details, we choose subtask 1: Aspect term extraction of the restaurants domain, and split 10% of the training data as the development set. Texts from datasets SemEval-2014 and SemEval-2015 are used as unlabeled data.

## Appendix F Experiment Results Details

This section contains detail experimental results for precision, recall and F1-value, see Table 8, 10,11, 12,13.

\begin{table}
\begin{tabular}{|l|c|c|} \hline Hyperparameters & Teacher & Student \\ \cline{2-3} Batch size & 32 & \{16,32,64,128\} \\ \hline Learning rate & Logit and BERT initial : le-s & \{1e-3,5e-4\} \\ \hline Dropout rate & Embedding : 0.5 & Embedding: (0.5,0.6) \\ \cline{2-3}  & BERT output: 0.5 & LSTM output: 0.5 \\ \hline LSTM hidden size & - & 300 \\ \hline \end{tabular}
\end{table}
Table 6: Hyperparameters for student and teacher models.

\begin{table}
\begin{tabular}{l l c c c c c c} \hline
**Language** & **Dataset** & **Wgt** & **Type** & **Train** & **Unlabeled** & **Dev** & **Test** \\ \hline \multirow{4}{*}{English} & \multicolumn{3}{c}{Oursates 5.0} & 18 & 55.98± & 36.06± & 8.5 & 8.3± \\  & & T & 1088± & 86.68± & 76.6± & 74.7± & 15.2± \\ \cline{2-7}  & CoNLL-2003 & 4 & S & 15.08± & 50.0± & 3.8± & 5.3± \\  & \multicolumn{3}{c}{T} & 2nd.04± & 188± & 45.8± & 45.4± & 46.4± \\ \cline{2-7}  & Semi-2016 Task 5 & 1 & 1 & 1.88± & 3.9± & 0.2± & 8.7± \\ \cline{2-7}  & \multicolumn{3}{c}{T} & 2rd.7± & 3rd.8± & 7.3± & 17.2± \\ \hline \multirow{4}{*}{Chinese} & MSRA & 3 & 5 & 46.4± & 25.8± & 4.4± \\  & \multicolumn{3}{c}{T} & 97.9± & 100± & 17.2± & 17.2± \\ \cline{2-7}  & Wicho & 4 & 5 & 7.4± & 50.0± & 9.8± & 0.7± \\ \cline{2-7}  & OotNets 4.0 & 4 & 7 & 13.8± & 211± & 4.0± & 14.8± \\ \hline \end{tabular}
\end{table}
Table 5: Summary statistics for the datasets. #Ent: number of entities; S: sentence size; T: token size.





\begin{table}
\begin{tabular}{l|c c c} \hline Model & P & R & F1 \\ \hline BERT+CRF (teacher) & 89.51 & 88.35 & 89.92 \\ Vanilla BiLSTM+CRF & 82.88 & 82.29 & 82.58 \\ + TE & 87.48 & 87.22 & 87.35 \\ + TE + TOKEN-EM & 87.41 & 87.37 & 87.39 \\ + TE + TOKEN-POS & 88.51 & 87.34 & 88.17 \\ + TE + SEQ & 88.72 & 87.96 & 88.34 \\ + TE + DA & 88.10 & 86.85 & 87.52 \\ + TE + DA + TOKEN-EM & 88.31 & 87.80 & 88.05 \\ + TE + DA + TOKEN-POS & 89.02 & 87.57 & 88.29 \\ + TE + DA + SEQ (student) & 89.51 & 88.31 & 88.91 \\ \hline \end{tabular}
\end{table}
Table 13: Results on OntoNotes 5.0.

\begin{table}
\begin{tabular}{l|c c c} \hline Model & P & R & F1 \\ \hline BERT+CRF (teacher) & 89.51 & 88.35 & 89.92 \\ Vanilla BiLSTM+CRF & 82.88 & 82.29 & 82.58 \\ + TE & 87.48 & 87.22 & 87.35 \\ + TE + TOKEN-EM & 87.41 & 87.37 & 87.39 \\ + TE + TOKEN-POS & 88.51 & 87.34 & 88.17 \\ + TE + SEQ & 88.72 & 87.96 & 88.34 \\ + TE + DA & 88.10 & 86.85 & 87.52 \\ + TE + DA + TOKEN-EM & 88.31 & 87.80 & 88.05 \\ + TE + DA + TOKEN-POS & 89.02 & 87.57 & 88.29 \\ + TE + DA + SEQ (student) & 89.51 & 88.31 & 88.91 \\ \hline \end{tabular}
\end{table}
Table 13: Results on OntoNotes 5.0.

\begin{table}
\begin{tabular}{l|c c c} \hline Model & P & R & F1 \\ \hline BERT+CRF (teacher) & 89.51 & 88.35 & 89.92 \\ Vanilla BiLSTM+CRF & 82.88 & 82.29 & 82.58 \\ + TE & 87.48 & 87.22 & 87.35 \\ + TE + TOKEN-EM & 87.41 & 87.37 & 87.39 \\ + TE + TOKEN-POS & 88.51 & 87.34 & 88.17 \\ + TE + SEQ & 88.72 & 87.96 & 88.34 \\ + TE + DA & 88.10 & 86.85 & 87.52 \\ + TE + DA + TOKEN-EM & 88.31 & 87.80 & 88.05 \\ + TE + DA + TOKEN-POS & 89.02 & 87.57 & 88.29 \\ + TE + DA + SEQ (student) & 89.51 & 88.31 & 88.91 \\ \hline \end{tabular}
\end{table}
Table 12: Results on CoNLL-2003.

\begin{table}
\begin{tabular}{l|c c c} \hline Model & P & R & F1 \\ \hline BERT+CRF (teacher) & 81.87 & 82.91 & 82.39 \\ Vanilla BiLSTM+CRF & 66.97 & 62.02 & 64.4 \\ + TE & 72.62 & 67.15 & 69.78 \\ + TE + TOKEN-EM & 72.13 & 68.85 & 70.46 \\ + TE + TOKEN-POS & 72.18 & 70.22 & 71.19 \\ + TE + SEQ & 75.34 & 69.55 & 72.33 \\ + TE + DA & 76.06 & 74.80 & 75.43 \\ + TE + DA + TOKEN-EM & 75.21 & 72.84 & 74.52 \\ + TE + DA + TOKEN-POS & 75.99 & 73.80 & 74.88 \\ + TE + DA + SEQ (student) & 77.94 & 74.23 & 76.05 \\ \hline \end{tabular}
\end{table}
Table 11: Results on OntoNotes 4.0.





\begin{table}
\begin{tabular}{l l|c c|c c c c} \hline  & \multicolumn{4}{c|}{**Models**} & \multicolumn{6}{c}{**Datasets**} \\ \cline{3-8}  & \multicolumn{2}{c|}{**English Datasets**} & \multicolumn{5}{c}{**Chinese Datasets**} \\ \hline
**English** & \multicolumn{1}{c|}{**Chinese**} & \multicolumn{1}{c|}{**CoNLL-2003****OntoNotes 5.0**} & \multicolumn{1}{c}{**MSRA**} & \multicolumn{1}{c}{**Weibo**} & \multicolumn{1}{c}{**OntoNotes 4.0**} \\ \hline BERT+CRF (teacher) & Vanilla BERT+CRF & NA & NA & 94.80 & 67.33 & 79.16 \\  & Glyce+BERT & NA & NA & 95.54 & 67.60 & 80.62 \\  & BERT+CRF (teacher) & **92.03** & **89.92** & **95.91** & **71.22** & **82.39** \\ \hline + BP (no distillation) & & 89.01 & 88.16 & 93.68 & 64.09 & 75.07 \\   + BP + TOKEN-EM & & 88.53 & 88.49 & 94.47 & 65.99 & 75.59 \\   + BP + TOKEN-POS & & 89.21 & 88.46 & 94.55 & 65.10 & 75.35 \\   + BP + SEQ & **89.81** & **88.60** & **94.62** & **66.42** & **77.53** \\ \hline + BP + DA (no distillation) & 90.86 & 88.59 & 94.83 & 70.11 & 79.17 \\   + BP + DA + TOKEN-EM & 90.83 & 88.67 & 94.69 & 70.09 & 79.35 \\   + BP + DA + TOKEN-POS & 91.04 & 88.78 & 94.84 & 69.73 & 79.31 \\   + BP + DA + SEQ (student) & **91.21** & **89.06** & **95.07** & **70.65** & **79.91** \\ \hline \end{tabular}
\end{table}
Table 9: Comparison of F1 scores on four transformer layers as student model. The abbreviations herein is as same as Table 1, except BP means BERT pretrained parameters are used. Noting that, we simply choose the first four transformer layers from BERT model to initialize the student model, however, other reasonable strategies as in [22, 2] will be studied in our future work.