SlowGuess/NAACL_2021_OCR huggingface dataset:
We briefly describe the modeling and training specifications choices for the teacher and student models below.
Teacher Model. To build a strong learner, our teacher model is constructed by a BERT model followed by a CRF layer, and we denote it as BERT+CRF. Specifically, we use  $\mathrm{BERT}_{\mathrm{BASE}}^1$  model as our feature encoder, which is known to perform strongly across a wide range of NLP tasks.
A dropout layer is concatenated to the BERT, followed by a fully connected layer that computes the  $\mathrm{logit}(\mathbf{x})$  for the labels at each location. We further apply an additional CRF-layer as defined in (2) to account for the temporal dependencies among the labels, similar to the work of Meng et al. (2019). The teacher model is trained using the standardized fine-tune paradigm (Devlin et al., 2019). Following Howard and Ruder (2018); Sun et al. (2019a), we set different learning rates for each layer. A larger rate is used for CRF, and for the BERT the learning rates decays by a factor of 0.9 as the layers approaches the input.
Student Model. For our student model, we want it to be light, fast yet still sufficiently expressive. To this end, we use the BiLSTM+CRF architecture proposed in Huang et al. (2015). This model exploits a Bidirectional LSTM to map input sequence  $\mathbf{x}$  into a sequence of feature vectors, which accounts for the context from both directions. The rest of the construction follows what has been described for the teacher model, with the BERT part replaced by the BiLSTM. We reuse the learned word-embeddings from the teacher model and keep it frozen during training. Empirically, we find this strategy produces better results, possibly due to reduced effort transferring the knowledge of richer embedding representation compared to alternatives such as word2vec, and it also avoids over-fitting.
5.3 Baselines, Variations and Evaluation
In addition to the vanilla teacher and student models described above, we also considered the following strong established NER baselines in our experiments.
- BERT teacher baselines (Chinese NER)  
BERT+Glyce (Meng et al., 2019).
- Non-BERT student baselines (English NER) LSTM-CNNs (Chiu and Nichols, 2016), LEX (Ghaddar and Langlais, 2018), IDCNN (Strubell et al., 2017b), HSCRF (Liu et al., 2019a).
Non-BERT student baselines (Chinese NER) SUL (Peng et al., 2019), Lattice-LSTM (Zhang and Yang, 2018), NMDM (Ding et al., 2019), ME-CNER (Xu et al., 2019).
Variations. To further understand the contributions from different components of our proposal, we run different variations of the model to dissect the gains. We use BiLSTM+CRF as our
Table 1: Comparison of F1 scores.  $\uparrow$  denotes the gain relative to Vanilla BiLSTM+CRF baseline. Best results in each category are shown in bold. Results for the competing baselines are collected from the original papers.
Models	Datasets
English Datasets	Chinese Datasets
English	Chinese	CoNLL-2003	OntoNotes 5.0	MSRA	Weibo	OntoNotes 4.0
BERT+CRF (teacher)	Vanilla BERT+CRF	NA	NA	94.80	67.33	79.16
Glyce+BERT	NA	NA	95.54	67.60	80.62
BERT+CRF (teacher)	92.03	89.92	95.91	71.22	82.39
LSTM-CNNs	Lattice-LSTM	90.91	86.17	93.18	58.79	73.88
IDCNN	NMDM	90.54	86.84	94.4	59.5	76.0
LEX	SUL	90.52	87.95	93.44	61.24	75.54
HSCRF	ME-CNER	91.10	89.94	91.45	68.93	NA
Vanilla BiLSTM+CRF	80.08	82.58	90.70	51.85	64.40
+ TE (no distillation)	88.35↑8.27	87.35↑4.77	91.69↑0.99	54.87↑3.02	69.78↑5.38
+ TE +TOKEN-EM	88.46↑8.38	87.39↑4.81	91.51↑0.81	53.55↑1.70	70.46↑6.06
+ TE +TOKEN-POS	88.66↑8.58	88.17↑5.59	91.77↑1.07	54.34↑2.49	71.19↑6.79
+ TE + SEQ	89.54↑9.46	88.34↑5.76	91.98↑1.98	57.14↑5.29	72.33↑7.93
+ TE + DA (no distillation)	90.69↑10.61	87.52↑4.94	92.68↑1.96	69.74↑17.89	74.53↑10.13
+ TE + DA +TOKEN-EM	90.49↑10.41	88.05↑5.47	92.66↑1.96	69.83↑17.98	74.52↑10.12
+ TE + DA +TOKEN-POS	90.98↑10.90	88.29↑5.71	92.59↑1.89	69.90↑18.05	74.88↑10.48
+ TE + DA + SEQ (student)	91.17↑11.09	88.91↑6.32	92.99↑2.29	71.62↑19.77	76.05↑11.65
base model, and consider variants with combinations of the following components in our experiments: (i) TE: use fixed pre-trained teacher embedding; (ii)TOKEN-EM: token-level distillation with loss  $\mathcal{L}_{\mathrm{Token - em}}$  ; (iii)TOKEN-POS: token-level distillation with with loss  $\mathcal{L}_{\mathrm{Token - pos}}$  ; (iv) SEQ: multi-grained sequence-level distillation with loss  $\mathcal{L}_{\mathrm{Seq - all}}^{\mathrm{u}}$  ; (v) DA: augmented with unlabeled data. We also vary the size of best candidate set from  $K = 1$  to  $K = 15$
Evaluation. We report the F1 score following Sang and De Meulder (2003), and relegate other quantitative metrics such as Precision and Recall to the supplementary material. We apply early stopping with max patience set to 5 based on the performance of the development set.
5.4 Analysis of Results
From Table 1 we first find out that, in all three Chinese datasets, our teacher model outperforms two baselines (Meng et al., 2019). We owe this to the layer-wise learning rate and discriminative finetuning strategies (Howard and Ruder, 2018; Sun et al., 2019a). Another analysis in terms of teacher is that directly copying the teacher embedding to the student model can be most helpful, for both English and Chinese datasets.
Regarding distillation, it achieved cross-theboard performance gains relative to the nodistillation TE baseline. Our final proposal, namely  $\mathrm{TE + DA + SEQ}$ , performs better or similarly to almost all of its non-BERT baselines. Note that these competing baselines have leveraged additional do
Table 2: Performance with different loss combinations.
Loss Combinations	Datasets
OntoNotes 4.0	CoNLL-2003
P	R	F1	P	R	F1
LSeq-all	77.96	74.23	76.05	90.85	91.50	91.17
+ Token-pos	76.63	74.47	75.53	90.63	91.51	91.07
-Seq-fuzzy	78.11	73.43	75.70	90.43	91.11	90.77
-Seq-ce	75.71	74.87	75.29	90.35	91.60	90.97
Table 3: Comparison of automated uncertainty weighting (auto) defined in (9) versus equal weighting (equal).
Datasets	Without DA	With DA
auto	equal	auto	equal
MSRA	91.98	91.77	92.99	92.83
Weibo	57.14	55.59	71.62	70.09
OntoNotes 4.0	72.32	71.32	76.05	75.83
CoNLL-2003	89.54	89.26	91.17	91.09
OntoNote 5.0	88.34	88.27	88.91	88.42
main knowledge, such as lexicon information, to boost NER performance (see Related Work). Such practice not only adds specialized modeling effort and complexity, but also make the resulting architecture less generalizable to other tasks. Our results show that we can match the performance via reaping the knowledge from more sophisticated models using general purpose sequence-level distillation, rather than appealing to dedicated modeling effort.
We also observed that inducing data augmentation consistently improves student learning. And notably, in all cases, the sequence-level distillation outperforms token-level distillation, especially in the absence of data augmentation.
Table 4: Comparison of efficiency.
Models	No. of parameters	Inference time (s)
Student	2.6M	1.5 × 10-3
Teacher	110.9M	7.0 × 10-2
Figure 4: Performance with different candidate size  $K$ .
Size of Candidate Set. In Figure 4, we compare how varying the size of candidate set affects performance. The performance peaks at a moderate  $K$ , after that the gain tapers off. This is because when using an excessive  $K$ , the distillation process starts to include more inaccurate label-sequences, compromising the learning efficiency (as top- $K$  label-sequences are treated equally in the fuzzy objective).
Effect of Augmenting with Unlabeled Data. In Figure 5, we report the F1 scores with different number of unlabeled instances to augment the distillation phase for on Weibo dateset, which has very few training instances but an enormous unlabeled set. The use of data augmentation drastically boosts student's performance (F1:  $57.14 \rightarrow 71.62$ ). The distillation performance monotonically increases as more unlabeled instances are used, and the performance closes, even beats the teacher model in the large sample limit.
Loss Combinations. We further explore additional combinations of losses to sharpen our understanding, with main results summarized in Table 2. Combinations of sequence-level and token-level loss (i.e.,  $\mathcal{L}_{\mathrm{Seq - all}}^{\mathrm{u}} + \mathcal{L}_{\mathrm{Token - pos}}$  ) reveal a slight drop in performance. We also subtract each individual  $\mathcal{L}_{\mathrm{Seq - fuzzy}}$  and  $\mathcal{L}_{\mathrm{Seq - ce}}$  from  $\mathcal{L}_{\mathrm{Seq - all}}^{\mathrm{u}}$ . It appears that the coarse-grained loss  $\mathcal{L}_{\mathrm{Seq - fuzzy}}$  is propitious to increase the recall value, while using the finegrained loss  $\mathcal{L}_{\mathrm{Seq - ce}}$  will get better precision value.
Effect of Uncertainty Weighting. In this paper, we balance our loss functions with different scales
Figure 5: Performance gains using more unlabeled data on the Weibo dataset. Dashed line denotes the teacher performance baseline.
by injecting uncertainty (Cipolla et al., 2018). This method yields a regularization term of weight, preventing the weight tends to zero. Table 3 shows that, regularized weighting outperforms equal weighting in all cases.
Computation Efficiency. To examine the efficiency gains, we compare inference time difference between the teacher and student model, as reported in Table 4. Our student model achieves over  $40 \times$  reduction and speedup, while achieving comparable performance compared with the teacher model. Alternative Choice of Student Model. We also exam the performance of multi-grained distillation method on another student model, that is four transformer layers, which has a similarity structure as BERT. The results are showed in supplementary material (Table 9).
In this work, we develop novel multi-grained knowledge distillation techniques to train a light NER model with comparable performance with their more sophisticated counterparts. In particular, we show that Viterbi algorithm can be exploited to impart the knowledge of  $k$ -best predictions from the teacher model to the student. We further advocated the use of CRF adjustments, fuzzy objective and data augmentation to improve performance. Our empirical experiments carefully analyze the gains from our proposal on a wide range of NER benchmarks, via efficiently transferring knowledge from a powerful BERT model to a much more compact BiLSTM student. In future work, we seek to extend the proposed framework to more general distillation applications where CRF is used, such as speech recognition, and distill with more general representation transfer schemes (Chen et al., 2020).
The authors would like to thank the anonymous reviewers for their insightful comments. The research conducted at Duke was supported in part by the DOE, NSF and ONR.
Tom B Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. 2020. Language models are few-shot learners. arXiv preprint arXiv:2005.14165.
Wanxiang Che, Mengqiu Wang, Christopher D Manning, and Ting Liu. 2013. Named entity recognition with bilingual constraints. In NAACL.
Junya Chen, Zidi Xiu, Benjamin Goldstein, Ricardo Henao, Lawrence Carin, and Chenyang Tao. 2020. Supercharging imbalanced data learning with causal representation transfer. arXiv preprint arXiv:2011.12454.
Jason P.C. Chiu and Eric Nichols. 2016. Named entity recognition with bidirectional LSTM-CNNs. Transactions of the Association for Computational Linguistics, 4:357-370.
R. Cipolla, Y. Gal, and A. Kendall. 2018. Multi-task learning using uncertainty to weigh losses for scene geometry and semantics. In CVPR.
Ronan Collobert, Jason Weston, Léon Bottou, Michael Karlen, Koray Kavukcuoglu, and Pavel Kuksa. 2011a. Natural language processing (almost) from scratch. Journal of machine learning research, 12(ArtICLE):2493-2537.
Ronan Collobert, Jason Weston, Léon Bottou, Michael Karlen, Koray Kavukcuoglu, and Pavel Kuksa. 2011b. Natural language processing (almost) from scratch. Journal of machine learning research, 12(Aug):2493-2537.
Linguistic Data Consortium et al. 2011. Ontonotes release 4.0 [online] available from: https://catalog.ldc.upenn.edu.LDC2011T03.
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language understanding. In *NAACL*.
Ruixue Ding, Pengjun Xie, Xiaoyan Zhang, Wei Lu, Linlin Li, and Luo Si. 2019. A neural multi-digraph model for chinese ner with gazetteers. In ACL.
Abbas Ghaddar and Philippe Langlais. 2018. Robust lexical features for improved neural network namedentity recognition. In COLING.
Yunchao Gong, Liu Liu, Ming Yang, and Lubomir Bourdev. 2014. Compressing deep convolutional networks using vector quantization. arXiv preprint arXiv:1412.6115.
Tao Gui, Ruotian Ma, Qi Zhang, Lujun Zhao, Yu-Gang Jiang, and Xuanjing Huang. 2019. Cnn-based chinese ner with lexicon rethinking. In IJCAI.
Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. 2015. Distilling the knowledge in a neural network. Computer Science, 14(7):38-39.
Jeremy Howard and Sebastian Ruder. 2018. Universal language model fine-tuning for text classification. In ACL.
Liang Huang and David Chiang. 2005. Better k-best parsing. In Proceedings of the Ninth International Workshop on Parsing Technology.
Zhiheng Huang, Yi Chang, Bo Long, Jean-Francois Crespo, Anlei Dong, Sathiya Keerthi, and Su-Lin Wu. 2012. Iterative Viterbi A* algorithm for k-best sequential decoding. In ACL.
Zhiheng Huang, Wei Xu, and Kai Yu. 2015. Bidirectional LSTM-CRF models for sequence tagging. CoRR, abs/1508.01991.
Xiaoqi Jiao, Yichun Yin, Lifeng Shang, Xin Jiang, Xiao Chen, Linlin Li, Fang Wang, and Qun Liu. 2019. Tinybert: Distilling bert for natural language understanding. arXiv preprint arXiv:1909.10351.
Yoon Kim and Alexander M. Rush. 2016. Sequence-level knowledge distillation. In EMNLP.
John Lafferty, Andrew Mccallum, and Fernando Pereira. 2001. Conditional random fields: Probabilistic models for segmenting and labeling sequence data. In ICML.
Gina-Anne Levow. 2006. The third international Chinese language processing bakeoff: Word segmentation and named entity recognition. In Proceedings of the Fifth SIGHAN Workshop on Chinese Language Processing.
Tianyu Liu, Jin-Ge Yao, and Chin-Yew Lin. 2019a. Towards improving neural named entity recognition with gazetteers. In ACL.
Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019b. Roberta: A robustly optimized bert pretraining approach. arXiv preprint arXiv:1907.11692.
Xindian Ma, Peng Zhang, Shuai Zhang, Nan Duan, Yuexian Hou, Dawei Song, and Ming Zhou. 2019. A tensorized transformer for language modeling. arXiv preprint arXiv:1906.09777.
JS McCarley. 2019. Pruning a bert-based question answering model. arXiv preprint arXiv:1910.06360.
Yuxian Meng, Wei Wu, Fei Wang, Xiaoya Li, Ping Nie, Fan Yin, Muyu Li, Qinghong Han, Xiaofei Sun, and Jiwei Li. 2019. Glyce: Glyph-vectors for Chinese character representations. In NIPS.
Subhabrata Mukherjee and Ahmed Hassan Awadallah. 2019. Distilling transformers into simple neural networks with unlabeled transfer data. arXiv preprint arXiv:1910.01769.
Raden Mu'az Mun'im, Nakamasa Inoue, and Koichi Shinoda. 2019. Sequence-level knowledge distillation for model compression of attention-based sequence-to-sequence speech recognition. In ICASSP.
Jesper Nielsen. 2011. A coarse-to-fine approach to computing the k-best viterbi paths. In CPM.
Minlong Peng, Ruotian Ma, Qi Zhang, and Xuanjing Huang. 2019. Simplify the usage of lexicon in Chinese ner. arXiv preprint arXiv:1908.05969.
Nanyun Peng and Mark Dredze. 2015. Named entity recognition for chinese social media with jointly trained embeddings. In EMNLP.
Sameer Pradhan, Alessandro Moschitti, Nianwen Xue, Hwee Tou Ng, Anders Björkelund, Olga Uryupina, Yuchen Zhang, and Zhi Zhong. 2013. Towards robust linguistic analysis using ontonotes. In CoNLL.
Erik F Sang and Fien De Meulder. 2003. Introduction to the conll-2003 shared task: Language-independent named entity recognition. arXiv preprint cs/0306050.
Jingbo Shang, Liyuan Liu, Xiaotao Gu, Xiang Ren, Teng Ren, and Jiawei Han. 2018. Learning named entity tagger using domain-specific dictionary. In EMNLP.
Suraj Srinivas and R. Venkatesh Babu. 2015. Data-free parameter pruning for deep neural networks. Computer Science, pages 2830-2838.
Emma Strubell, Patrick Verga, David Belanger, and Andrew McCallum. 2017a. Fast and accurate entity recognition with iterated dilated convolutions. In EMNLP.
Emma Strubell, Patrick Verga, David Belanger, and Andrew McCallum. 2017b. Fast and accurate entity recognition with iterated dilated convolutions. In EMNLP.
Chi Sun, Xipeng Qiu, Yige Xu, and Xuanjing Huang. 2019a. How to fine-tune bert for text classification? arXiv preprint arXiv:1905.05583.
Siqi Sun, Yu Cheng, Zhe Gan, and Jingjing Liu. 2019b. Patient knowledge distillation for bert model compression. arXiv preprint arXiv:1908.09355.
Yu Sun, Shuohuan Wang, Yukun Li, Shikun Feng, Xuyi Chen, Han Zhang, Xin Tian, Danxiang Zhu, Hao Tian, and Hua Wu. 2019c. Ernie: Enhanced representation through knowledge integration. arXiv preprint arXiv:1904.09223.
Jiaxi Tang, Rakesh Shivanna, Zhe Zhao, Dong Lin, Anima Singh, Ed H. Chi, and Sagar Jain. 2020. Understanding and Improving Knowledge Distillation. arXiv:2002.03532 [cs, stat]. ArXiv:2002.03532.
Raphael Tang, Yao Lu, Linqing Liu, Lili Mou, Olga Vechtomova, and Jimmy Lin. 2019. Distilling task-specific knowledge from bert into simple neural networks. arXiv preprint arXiv:1903.12136.
Henry Tsai, Jason Riesa, Melvin Johnson, Naveen Arivazhagan, Xin Li, and Amelia Archer. 2019. Small and Practical BERT Models for Sequence Labeling. In EMNLP-IJCNLP.
Iulia Turc, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. Well-read students learn better: On the importance of pre-training compact models. arXiv preprint arXiv:1908.08962.
Andrew Viterbi. 1967. Error bounds for convolutional codes and an asymptotically optimum decoding algorithm. IEEE transactions on Information Theory, 13(2):260-269.
Xinyu Wang, Yong Jiang, Nguyen Bach, Tao Wang, Fei Huang, and Kewei Tu. 2020. Structure-level knowledge distillation for multilingual sequence labeling. In ACL.
Jiaxiang Wu, Cong Leng, Yuhang Wang, Qinghao Hu, and Jian Cheng. 2016. Quantized convolutional neural networks for mobile devices. In CVPR.
Canwen Xu, Feiyang Wang, Jialong Han, and Chenliang Li. 2019. Exploiting multiple embeddings for chinese named entity recognition. In CIKM.
Jie Yang and Yue Zhang. 2018. NCRF++: An open-source neural sequence labeling toolkit. In ACL.
Zhilin Yang, Zihang Dai, Yiming Yang, Jaime Carbonell, Ruslan Salakhutdinov, and Quoc V Le. 2019. Xlnet: Generalized autoregressive pretraining for language understanding. arXiv preprint arXiv:1906.08237.
Xiyu Yu, Tongliang Liu, Xinchao Wang, and Dacheng Tao. 2017. On compressing deep models by low rank and sparse decomposition. In CVPR.
Yue Zhang and Jie Yang. 2018. Chinese NER using lattice LSTM. In ACL.
A Datasets statistic
Detailed summary statistics of the datasets.
Table 5: Summary statistics for the datasets. #Ent: number of entities; S: sentence size; T: token size.
Language	Dataset	#Ent	Type	Train	Unlabeled	Dev	Test
English	OntoNotes 5.0	18	S	59.9k	36.0k	8.5k	8.3k
T	1088.5k	676.6k	147.7k	152.7k
CoNLL-2003	4	S	15.0k	50.0k	3.3k	3.5k
T	204.6k	1485.4k	51.4k	46.4k
SemEval-2016 Task5	1	S	1.8k	3.9k	0.2k	0.7k
T	27.6k	78.8k	3.7k	12.6k
Chinese	MSRA	3	S	46.4k	25.5k	-	4.4k
T	979.2k	1190.2k	-	172.6k
Weibo	4	S	1.4k	50.0k	0.3k	0.3k
T	73.5k	2118.9k	14.4k	14.8k
OntoNotes 4.0	4	S	15.7k	21.3k	4.3k	4.3k
T	491.9k	659.0k	200.5k	208.1k
In Table 6 we list the search grids for our hyperparameter tuning. For each dataset we evaluate all combinations and report the test set results for best-performing model on the validation set.
Table 6: Hyperparameters for student and teacher models.
Hyperparameters	Teacher	Student
Batch size	32	{16,32,64,128}
Learning rate	Logit and BERT initial: le-5	{1e-3,5e-4}
CRF: le-3
Dropout rate	Embedding: 0.5	Embedding: {0.5,0.6}
BERT output: 0.5	LSTM output: 0.5
LSTM hidden size	-	300
C  $k$ -best Viterbi Algorithm
We implemented the  $k$ -best Viterbi algorithm generalized from the classic Viterbi algorithm, that is to storage the Top-K rather than maximum scores at each time step and tag type. Though exists more efficient  $k$ -best Viterbi implementation (Huang et al., 2012) for large label scenarios, it only offers marginal efficiency gains as in our tasks. Unlike existing work (Yang and Zhang, 2018), we remove short examples where  $K$  is larger than the possible number of label sequences. In our application, we also output the probability rather than the path score, and this needs another dynamic programming to calculate the denominator in (1). The transition matrix is rule-constrained such that invalid transitions (e.g.,  $\mathsf{B - ORG}\to \mathsf{I - PER}$ ) are prohibited.
D Results on student with transformer structure
Though our main result is reported as the BiLSTM student, the multi-grained distillation method proposed in this article is not restricted to this student model. We show herein the experiment results (Table 9) on the four transformer layers as student model.
Algorithm 1: Pseudocode of  $k$ -best Viterbi Algorithm
Input: The length of input sequence  $\mathbf{x}^{(i)}$  as  $L$ , the number of tags  $T$ , transition matrix  $A[T\times T]$ , the logits  $[L\times T]$  of  $\mathbf{x}^{(i)}$ ,  $K$
Output:  $k$ -best paths and corresponding probabilities
1 Initialize:CurrentPath  $= 1:T$
2 Initialize:CurrentScore  $=$  logits[0,  $\vdots$
3 for  $m = 0$  to  $L - 1$  do
4  $w =$  CurrentPath.shape[1]; 5 Update CurrentPath by adding  $T$  possible tags on the end of every path;
6 Update CurrentScore of each path through (2);
7 if CurrentPath.shape[0]  $<  K\times T$  then
8 Continue
10 Group sort  $k$  -best on CurrentScore;
11 Update CurrentPath using the indexes of  $k$  -best scores, resulting  $[w + 1,K\times T]$
12 Update CurrentScore using the indexes of  $k$  -best scores, resulting  $[K,1]$
13 end
14 Calculate the denominator in (1) using another dynamic programming, for details see (Collobert et al., 2011b);
15 return  $k$ -best paths and  $k$ -best probabilities or
$$
\left\{\left(\mathbf {y} _ {1} ^ {(i)}, p _ {1} ^ {(i, \mathbf {t})}\right), \dots , \left(\mathbf {y} _ {K} ^ {(i)}, p _ {K} ^ {(i, \mathbf {t})}\right) \right\},
$$
E Results on Sentiment Analysis Task
Table 7 shows the experimental results on dataset SemEval-2016 Task 5 of aspect based sentiment analysis. In details, we choose subtask 1: Aspect term extraction of the restaurants domain, and split  $10\%$  of the training data as the development set. Texts from datasets SemEval-2014 and SemEval-2015 are used as unlabeled data.
Table 7: Results on SemEval-2016 Task 5.
Model	P	R	F1
BERT+CRF (teacher)	80.42	79.90	80.16
Vanilla BiLSTM+CRF	76.70	65.11	70.43
+ TE	74.86	66.08	71.20
+ TE +TOKEN-EM	73.69	68.49	71.12
+ TE +TOKEN-POS	76.33	64.79	70.09
+ TE + SEQ	77.02	67.36	71.87
+ TE + DA	77.08	67.04	71.71
+ TE + DA +TOKEN-EM	68.04	74.60	71.17
+ TE + DA +TOKEN-POS	69.59	72.83	71.17
+ TE + DA + SEQ (student)	75.26	70.90	73.01
F Experiment Results Details
This section contains detail experimental results for precision, recall and F1-value, see Table 8, 10,11, 12,13.
Table 8: Results on MSRA.
Model	P	R	F1
BERT+CRF (teacher)	95.96	95.86	95.91
Vanilla BiLSTM+CRF	91.57	89.84	90.70
+ TE	93.06	90.88	91.96
+ TE +TOKEN-EM	92.37	90.67	91.51
+ TE +TOKEN-POS	93.23	90.36	91.77
+ TE + SEQ	93.26	90.73	91.98
+ TE + DA	93.77	91.61	92.68
+ TE + DA +TOKEN-EM	93.77	91.57	92.66
+ TE + DA +TOKEN-POS	93.61	91.59	92.59
+ TE + DA + SEQ (student)	93.90	92.11	92.99
Table 13: Results on OntoNotes 5.0.
Model	P	R	F1
BERT+CRF (teacher)	89.51	88.35	89.92
Vanilla BiLSTM+CRF	82.88	82.29	82.58
+ TE	87.48	87.22	87.35
+ TE +TOKEN-EM	87.41	87.37	87.39
+ TE +TOKEN-POS	88.51	87.34	88.17
+ TE + SEQ	88.72	87.96	88.34
+ TE + DA	88.10	86.85	87.52
+ TE + DA +TOKEN-EM	88.31	87.80	88.05
+ TE + DA +TOKEN-POS	89.02	87.57	88.29
+ TE + DA + SEQ (student)	89.51	88.31	88.91