SlowGuess/ACL_2020_OCR huggingface dataset:
Bi-LSTM is a bidirectional LSTM (Schuster and Paliwal, 1997) with GloVe (Pennington et al., 2014) embedding for slot tagging. It is trained on the support set and tested on the query samples.
SimBERT is a model that predicts labels according to cosine similarity of word embedding of non-fine-tuned BERT. For each word  $x_{j}$ , SimBERT finds its most similar word  $x_{k}^{\prime}$  in support set, and the label of  $x_{j}$  is predicted to be the label of  $x_{k}^{\prime}$ .
TransferBERT is a domain transfer model with the NER setting of BERT (Devlin et al., 2019). We pretrain the it on source domains and select the best model on the same dev set of our model. We deal with label mismatch by only transferring bottleneck feature. Before testing, we fine-tune it on target domain support set. Learning rate is set as 1e-5 in training and fine-tuning.
WarmProtoZero (WPZ) (Fritzler et al., 2019) is a few-shot sequence labeling model that regards sequence labeling as classification of every single word. It pre-trains a prototypical network (Snell et al., 2017) on source domains, and utilize it to do word-level classification on target domains without training. Fritzler et al. (2019) use randomly initialized word embeddings. To eliminate the influence of different embedding methods, we further implement WPZ with the pre-trained embedding of GloVe (Pennington et al., 2014) and BERT.
Matching Network (MN) is similar to WPZ. The only difference is that we employ the matching network (Vinyals et al., 2016) with BERT embedding for classification.
Results of 1-shot Setting Table 2 shows the 1-shot slot tagging results. Each column respectively shows the F1 scores of taking a certain domain as target domain (test) and use others as source domain (train & dev). As shown in the tables, our L-TapNet+CDT achieves the best performance. It outperforms the strongest few-shot learning baseline WPZ+BERT by average F1 scores of 14.64.
Our model significantly outperforms Bi-LSTM and TransferBERT, indicating that the number of labeled data under the few-shot setting is too scarce for both conventional machine learning and transfer
4For each episode, we calculate the F1 score on query samples with conlleval script: https://www.clips.uantwerpen.be/conl12000/ chunking/conlleval.txt
Model	1-shot Slot Tagging	Ave.
We	Mu	Pl	Bo	Se	Re	Cr
Bi-LSMT	10.36	17.13	17.52	53.84	18.44	22.56	8.64	21.21
SimBERT	36.10	37.08	35.11	68.09	41.61	42.82	23.91	40.67
TransferBERT	55.82	38.01	45.65	31.63	21.96	41.79	38.53	39.06
MN	21.74	10.68	39.71	58.15	24.21	32.88	69.66	36.72
WPZ	4.53	7.43	14.43	39.15	11.69	7.78	10.09	13.59
WPZ+GloVe	17.92	22.37	19.90	42.61	22.30	22.79	16.75	23.52
WPZ+BERT	46.72	40.07	50.78	68.73	60.81	55.58	67.67	55.77
TapNet	51.12	40.65	48.41	77.50	49.77	54.79	61.39	54.80
TapNet+CDT	66.30	55.93	57.55	83.32	64.45	65.65	67.91	65.87
L-WPZ+CDT	71.23	47.38	59.57	81.98	69.83	66.52	62.84	65.62
L-TapNet+CDT	71.53	60.56	66.27	84.54	76.27	70.79	62.89	70.41
Table 2: F1 scores on 1-shot slot tagging. +CDT denotes collapsed dependency transfer. Score below mid-line are from our methods, which achieve the best performance. Ave. shows the averaged scores. Results with standard deviations is showed in Appendix D.
Model	5-shots Slot Tagging	Ave.
We	Mu	Pl	Bo	Se	Re	Cr
Bi-LSMT	25.17	39.80	46.13	74.60	53.47	40.35	25.10	43.52
SimBERT	53.46	54.13	42.81	75.54	57.10	55.30	32.38	52.96
TransferBERT	59.41	42.00	46.07	20.74	28.20	67.75	58.61	46.11
MN	36.67	33.67	52.60	69.09	38.42	33.28	72.10	47.98
WPZ	9.54	14.23	18.12	44.65	18.98	12.03	14.05	18.80
WPZ+GloVe	26.61	34.25	22.11	50.55	28.53	34.16	23.69	31.41
WPZ+BERT	67.82	55.99	46.02	72.17	73.59	60.18	66.89	63.24
TapNet	53.03	49.80	54.90	83.36	63.07	59.84	67.02	61.57
TapNet+CDT	66.48	66.36	68.23	85.76	73.60	64.20	68.47	70.44
L-WPZ+CDT	74.68	56.73	52.20	78.79	80.61	69.59	67.46	68.58
L-TapNet+CDT	71.64	67.16	75.88	84.38	82.58	70.05	73.41	75.01
Table 3: F1 score results on 5-shots slot tagging. Our methods achieve the best performance. Results with standard deviations is showed in Appendix D.
learning models. Moreover, the performance of SimBERT demonstrates the superiority of metric-based methods over conventional machine learning models in the few-shot setting.
The original WarmProtoZero (WPZ) model suffers from the weak representation ability of its word embeddings. When we enhance it with GloVe and BERT word embeddings, its performance improves significantly. This shows the importance of embedding in the few-shot setting. Matching Network (MN) performs poorly in both settings. This is largely due to the fact that MN pays attention to all support word equally, which makes it vulnerable to the unbalanced amount of O-labels.
More specifically, those models that are fine-tuned on support set, such as Bi-LSTM and TransferBERT, tend to predict tags randomly. Those systems can only handle the cases that are easy to generalize from support examples, such as tags for proper noun tokens (e.g. city name and time). This shows that fine-tuning on extremely limited examples leads to poor generalization ability and
undertrained classifier. And for those metric based methods, such as WPZ and MN, label prediction is much more reasonable. However, these models are easy to be confused by similar labels, such as current_location and geographic poi. It indicates the necessity of well-separated label representations. Also illegal label transitions are very common, which can be well tackled by the proposed collapsed dependency transfer.
To eliminate unfair comparisons caused by additional information in label names, we propose the L-WPZ+CDT by enhancing the WarmProtoZero (WPZ) model with label name representation same to L-TapNet and incorporating it into the proposed CRF framework. It combines label name embedding and prototype as each label representation. Its improvements over WPZ mainly come from label semantics, collapsed dependency transfer and pair-wise embedding. L-TapNet+CDT outperforms L-WPZ+CDT by 4.79 F1 scores demonstrating the effectiveness of embedding projection. When compared with TapNet+CDT, L-TapNet+CDT achieves
an improvement of 4.54 F-score on average, which shows that considering label semantics and prototype helps improve emission score calculation.
Results of 5-shots Setting Table 3 shows the results of 5-shots experiments, which verify the proposed model's generalization ability in more shots situations. The results are consistent with 1-shot setting in general trending.
Ablation Test To get further an understanding of each component in our method (L-TapNet+CDT), we conduct ablation analysis on both 1-shot and 5-shots setting in Table 4. Each component of our method is removed respectively, including: collapsed dependency transfer, pair-wise embedding, label semantic, and prototype reference.
When collapsed dependency transfer is removed, we directly predict labels with emission score and huge F1 score drops are witnessed in all settings. This ablation demonstrates a great necessity for considering label dependency.
For our method without pair-wise embedding, we represent query and support sentences independently. We address the drop to the fact that support sentences can provide domain-related context, and pair-wise embedding can leverage such context and provide domain-adaptive representation for words in query sentences. This helps a lot when computing a word's similarity to domain-specific labels.
When we remove the label-semantic from L-TapNet, the model degenerates into TapNet+CDT enhanced with prototype in emission score. The drops in results show that considering label name can provide better label representation and help to model word-label similarity. Further, we also tried to remove the inner and beginning words in label representation and observe a 0.97 F1-score drop on 1-shot SNIPS. It shows that distinguishing B-I labels in label semantics can help tagging.
And if we calculate emission score without the prototype reference, the model loses more performance in 5-shots setting. This meets the intuition that prototype allows model to benefit more from the increase of support shots, as prototypes are directly derived from the support set.
Analysis of Collapsed Dependency Transfer While collapsed dependency transfer (CDT) brings significant improvements, two natural questions arise: whether CDT just learns simple transition rules and why it works.
Model	1-shot	5-shots
Ours	70.41	75.01
- dependency transfer	-10.01	-8.08
- pair-wise embedding	-8.29	-7.74
- label semantic	-9.57	-4.87
- prototype reference	-1.73	-3.33
Table 4: Ablation test over different components on slot tagging task. Results are averaged F1-score of all domains.
Model	1-shot	5-shots
L-TapNet	60.40	66.93
L-TapNet+Rule	65.30	69.64
L-TapNet+CDT	70.41	75.01
Table 5: Comparison between transition rules and collapsed dependency transfer (CDT).
To answer the first question, we replace CDT with transition rules in Table 5, which shows CDT can bring more improvements than transition rules.
To have a deeper insight into the effectiveness of CDT, we conduct an accuracy analysis of it. We assess the label predicting accuracy of different types of label bi-grams. The result is shown in Table 6. We further summarize the bi-grams into 2 categories: Border includes the bi-grams across the border of a slot span; Inner is the bi-grams within a slot span. We argue that improvements of Inner show successful reduction of illegal label transition from CDT. Interestingly, we observe that CDT also brings improvements by correctly predict the first and last token of a slot span. The results of Border verified our observation that CDT may helps to decide the boundaries of slot spans more accurately, which is hard to achieve by adding transition rules.
Traditional few-shot learning methods depend highly on hand-crafted features (Fei-Fei, 2006; Fink, 2005). Classical methods primarily focus on metric learning (Snell et al., 2017; Vinyals et al., 2016), which classifies an item according to its similarity to each class's representation. Recent efforts (Lu et al., 2018; Schwartz et al., 2019) propose to leverage the semantics of class name to enhance class representation. However, different from us, these methods focus on image classification where effects of name semantic are implicit and label dependency is not required.
Few-shot learning in natural language process
Transition Rule: We greedily predict the label for each word and block the result that conflicts with previous label.
Bi-gram Type	Proportion	L-TapNet	+CDT
Border	O-O	28.5%	82.7%	83.7%
O-B	24.5%	78.3%	81.5%
B-O	8.2%	72.4%	74.8%
I-O	5.8%	76.7%	81.7%
I-B/B-B	7.8%	65.0%	72.5%
Inner	B-I	13.3%	78.5%	83.6%
I-I	12.1%	77.8%	82.7%
Table 6: Accuracy analysis of label prediction on 1-shot slot tagging. The table shows accuracy and proportion of different bi-gram types in dataset.
ing has been explored for classification tasks, including text classification (Sun et al., 2019; Geng et al., 2019; Yan et al., 2018; Yu et al., 2018), entity relation classification (Lv et al., 2019; Gao et al., 2019; Ye and Ling, 2019), and dialog act prediction (Vlasov et al., 2018). However, few-shot learning for slot tagging is less investigated. Luo et al. (2018) investigated few-shot slot tagging using additional regular expressions, which is not comparable to our model due to the usage of regular expressions. Fritzler et al. (2019) explored few-shot named entity recognition with the Prototypical Network, which has a similar setting to us. Compared to it, our model achieves better performance by considering both label dependency transferring and label name semantics. Zero-shot slot tagging methods (Bapna et al., 2017; Lee and Jha, 2019; Shah et al., 2019) share a similar idea to us in using label name semantics, but has a different setting as few-shot methods are additionally supported by a few labeled sentences. Chen et al. (2016) investigate using label name in intent detection. In addition to learning directly from limited example, another research line of solving data scarcity problem in NLP is data augmentation (Fader et al., 2013; Zhang et al., 2015; Liu et al., 2017). For data augmentation of slot tagging, sentence generation based methods are explored to create additional labeled samples (Hou et al., 2018; Shin et al., 2019; Yoo et al., 2019).
In this paper, we propose a few-shot CRF model for slot tagging of task-oriented dialogue. To compute transition score under few-shot setting, we propose the collapsed dependency transfer mechanism, which transfers the prior knowledge of the label dependencies across domains with different label sets. And we propose L-TapNet to calculate emission score, which improves label representation with label name semantics. Experiment results
validate that both the collapsed dependency transfer and L-TapNet can improve the tagging accuracy.
We sincerely thank Ning Wang and Jiafeng Mao for the help on both paper and experiments. We are grateful for the helpful comments and suggestions from the anonymous reviewers. This work was supported by the National Natural Science Foundation of China (NSFC) via grant 61976072, 61632011 and 61772153.
Ankur Bapna, Gokhan Tur, Dilek Hakkani-Tur, and Larry Heck. 2017. Towards zero-shot frame semantic parsing for domain scaling. arXiv preprint arXiv:1707.02363.
Yun-Nung Chen, Dilek Hakkani-Tür, and Xiaodong He. 2016. Zero-shot learning of intent embeddings for expansion by convolutional deep structured semantic models. In Proc. of the ICASSP, pages 6045-6049. IEEE.
Alice Coucke, Alaa Saade, Adrien Ball, Theodore Bluche, Alexandre Caulier, David Leroy, Clément Doumouro, Thibault Gisselbrecht, Francesco Caltagirone, Thibaut Lavril, Maël Primet, and Joseph Dureau. 2018. Snips voice platform: an embedded spoken language understanding system for private-by-design voice interfaces. CoRR, abs/1805.10190.
Leon Derczynski, Eric Nichols, Marieke van Erp, and Nut Limsopatham. 2017. Results of the wnut2017 shared task on novel and emerging entity recognition. In Proc. of the 3rd Workshop on Noisy User-generated Text, pages 140-147.
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: pre-training of deep bidirectional transformers for language understanding. In Proc. of the NAACL-HLT, Volume I (Long and Short Papers), pages 4171-4186.
Anthony Fader, Luke Zettlemoyer, and Oren Etzioni 2013. Paraphrase-driven learning for open question answering. In Proc. of the ACL.
Li Fei-Fei. 2006. Knowledge transfer in learning to recognize visual objects classes. In International Conference on Development and Learning, pages 1-8.
Li Fei-Fei, Rob Fergus, and Pietro Perona. 2006. One-shot learning of object categories. IEEE transactions on pattern analysis and machine intelligence, 28(4):594-611.
Michael Fink. 2005. Object classification from a single example utilizing class relevance metrics. In NIPS pages 449-456.
G David Forney. 1973. The viterbi algorithm. Proc. of the IEEE, 61(3):268-278.
Alexander Fritzler, Varvara Logacheva, and Maksim Kretov. 2019. Few-shot classification in named entity recognition task. In Proc. of the SAC, pages 993-1000.
Tianyu Gao, Xu Han, Ruobing Xie, Zhiyuan Liu, Fen Lin, Leyu Lin, and Maosong Sun. 2019. Neural snowball for few-shot relation learning. arXiv preprint arXiv:1908.11007.
Ruiying Geng, Binhua Li, Yongbin Li, Xiaodan Zhu, Ping Jian, and Jian Sun. 2019. Induction networks for few-shot text classification. In Proc. of the EMNLP-IJCNLP, pages 3895-3904.
Yutai Hou, Yijia Liu, Wanxiang Che, and Ting Liu. 2018. Sequence-to-sequence data augmentation for dialogue language understanding. In Proceedings of the 27th International Conference on Computational Linguistics, COLING 2018, Santa Fe, New Mexico, USA, August 20-26, 2018, pages 1234-1245.
Jeremy Howard and Sebastian Ruder. 2018. Universal language model fine-tuning for text classification. In Proc. of the ACL, Volume 1: Long Papers, pages 328-339.
Zhiheng Huang, Wei Xu, and Kai Yu. 2015. Bidirectional lstm-crf models for sequence tagging. arXiv preprint arXiv:1508.01991.
Diederik P. Kingma and Jimmy Ba. 2015. Adam: A method for stochastic optimization. In Proc. of the ICLR.
John D. Lafferty, Andrew McCallum, and Fernando C. N. Pereira. 2001. Conditional random fields: Probabilistic models for segmenting and labeling sequence data. In Proc. of the ICML, ICML '01, pages 282-289, San Francisco, CA, USA. Morgan Kaufmann Publishers Inc.
Brenden M Lake, Ruslan Salakhutdinov, and Joshua B Tenenbaum. 2015. Human-level concept learning through probabilistic program induction. Science, 350(6266):1332-1338.
Sungjin Lee and Rahul Jha. 2019. Zero-shot adaptive transfer for conversational language understanding. In Proc. of the AAAI, volume 33, pages 6642-6649.
Ting Liu, Yiming Cui, Qingyu Yin, Wei-Nan Zhang, Shijin Wang, and Guoping Hu. 2017. Generating and exploiting large-scale pseudo training data for zero pronoun resolution. In Proc. of the ACL, pages 102-111.
Zhiwu Lu, Jiechao Guan, Aoxue Li, Tao Xiang, An Zhao, and Ji-Rong Wen. 2018. Zero and few shot learning with semantic feature synthesis and competitive learning. arXiv preprint arXiv:1810.08332.
Bingfeng Luo, Yansong Feng, Zheng Wang, Songfang Huang, Rui Yan, and Dongyan Zhao. 2018. Marrying up regular expressions with neural networks: A case study for spoken language understanding. In Proc. of the ACL, Volume 1: Long Papers, pages 2083-2093.
Xin Lv, Yuxian Gu, Xu Han, Lei Hou, Juanzi Li, and Zhiyuan Liu. 2019. Adapting meta knowledge graph information for multi-hop reasoning over few-shot relations. arXiv preprint arXiv:1908.11513.
Xuezhe Ma and Eduard H. Hovy. 2016. End-to-end sequence labeling via bi-directional lstm-cnns-crf. In Proc. of the ACL, Volume 1: Long Papers.
Erik G Miller, Nicholas E Matsakis, and Paul A Viola. 2000. Learning from one example through shared densities on transforms. In Proc. of the CVPR, volume 1, pages 464-471. IEEE.
Jeffrey Pennington, Richard Socher, and Christopher Manning. 2014. Glove: Global vectors for word representation. In Proc. of the EMNLP, pages 1532-1543.
Sameer Pradhan, Alessandro Moschitti, Nianwen Xue, Hwee Tou Ng, Anders Björkelund, Olga Uryupina, Yuchen Zhang, and Zhi Zhong. 2013. Towards robust linguistic analysis using onthonotes. In Proc. of the CoNLL, pages 143-152.
Nils Reimers and Iryna Gurevych. 2017. Reporting score distributions makes a difference: Performance study of lstm-networks for sequence tagging. In Proc. of the EMNLP.
Erik F. Tjong Kim Sang and Fien De Meulder. 2003. Introduction to the conll-2003 shared task: Language-independent named entity recognition. In Proc. of the CoNLL-HLT-NAACL, pages 142-147.
Ruhi Sarikaya, Paul A Crook, Alex Marin, Minwoo Jeong, Jean-Philippe Robichaud, Asli Celikyilmaz, Young-Bum Kim, Alexandre Rochette, Omar Zia Khan, Xiaohu Liu, et al. 2016. An overview of end-to-end language understanding and dialog management for personal digital assistants. In 2016 IEEE Spoken Language Technology Workshop (SLT), pages 391-397. IEEE.
Mike Schuster and Kuldip K. Paliwal. 1997. Bidirectional recurrent neural networks. IEEE Trans. Signal Processing, 45(11):2673-2681.
Eli Schwartz, Leonid Karlinsky, Rogerio Feris, Raja Giryes, and Alex M Bronstein. 2019. Baby steps towards few-shot learning with multiple semantics. arXiv preprint arXiv:1906.01905.
Darsh J. Shah, Raghav Gupta, Amir A. Fayazi, and Dilek Hakkani-Tür. 2019. Robust zero-shot cross-domain slot filling with example values. In Proc. of the ACL, Volume 1: Long Papers, pages 5484-5490.
Y. Shin, K. M. Yoo, and S. Lee. 2019. Utterance generation with variational auto-encoder for slot filling in spoken language understanding. IEEE Signal Processing Letters, 26(3):505-509.
Jake Snell, Kevin Swersky, and Richard Zemel. 2017. Prototypical networks for few-shot learning. In NIPS, pages 4077-4087.
Shengli Sun, Qingfeng Sun, Kevin Zhou, and Tengchao Lv. 2019. Hierarchical attention prototypical networks for few-shot text classification. In Proc. of the EMNLP-IJCNLP, pages 476-485.
Flood Sung, Yongxin Yang, Li Zhang, Tao Xiang, Philip HS Torr, and Timothy M Hospedales. 2018. Learning to compare: Relation network for few-shot learning. In Proc. of the CVPR, pages 1199-1208.
Gokhan Tur and Renato De Mori. 2011. Spoken language understanding: Systems for extracting semantic information from speech. John Wiley &amp; Sons.
Oriol Vinyals, Charles Blundell, Timothy Lillicrap, Daan Wierstra, et al. 2016. Matching networks for one shot learning. In NIPS, pages 3630-3638.
Vladimir Vlasov, Akela Drissner-Schmid, and Alan Nichol. 2018. Few-shot generalization across dialogue tasks. arXiv preprint arXiv:1811.11707.
Leiming Yan, Yuhui Zheng, and Jie Cao. 2018. Few-shot learning for short text classification. Multimedia Tools and Applications, pages 1-12.
Zhi-Xiu Ye and Zhen-Hua Ling. 2019. Multi-level matching and aggregation network for few-shot relation classification. In Proc. of the ACL, Volume 1: Long Papers, pages 2872-2881.
Kang Min Yoo, Youhyun Shin, and Sang-goo Lee. 2019. Data augmentation for spoken language understanding via joint variational generation. In Proc. of the AAAI, volume 33, pages 7402-7409.
Sung Whan Yoon, Jun Seo, and Jaekyun Moon. 2019. Tapnet: Neural network augmented with task-adaptive projection for few-shot learning. In Proc. of the ICML, pages 7115-7123.
Steve Young, Milica Gašić, Blaise Thomson, and Jason D Williams. 2013. Pomdp-based statistical spoken dialog systems: A review. Proc. of the IEEE, 101(5):1160-1179.
Mo Yu, Xiaoxiao Guo, Jinfeng Yi, Shiyu Chang, Saloni Potdar, Yu Cheng, Gerald Tesauro, Haoyu Wang, and Bowen Zhou. 2018. Diverse few-shot text classification with multiple metrics. In Proc. of the NAACL-HLT, Volume 1 (Long Papers), pages 1206-1215.
Amir Zeldes. 2017. The gum corpus: creating multilayer resources in the classroom. *Language Resources and Evaluation*, 51(3):581-612.
Xiang Zhang, Junbo Zhao, and Yann LeCun. 2015. Character-level convolutional networks for text classification. In Proc. of the NIPS, pages 649-657.
A Detail of Dataset
Table 7 shows the statistics of the original dataset used to construct few-shot experiment data.
Task	Dataset	Domain	# Sent	# Labels
Slot Tagging	Snips	We	2,100	10
Mu	2,100	10
Pl	2,042	6
Bo	2056	8
Se	2,059	8
Re	2,073	15
Cr	2,054	3
NER	CoNLL	News	20679	5
GUM	WiKi	3,493	12
WNUT	Social	5,657	7
OntoNotes	Mixed	159,615	19
B Few-shot experiments for Name entity recognition
Name entity recognition (NER) that identify predefined name entities, such as the person names, organizations and locations, can be modeled as a slot tagging task. Also, the data scarcity problem for a new domain exists in the NER task. For the above reasons, we conduct few-shot NER experiments to test our model's generation ability.
Table 7: Statistic of Original Dataset
Domain	1-shot	5-shots
Ave. |S|	Samples	Ave. |S|	Samples
News	3.38	4,000	15.58	1,000
Wiki	6.50	4,000	27.81	1,000
Social	5.48	4,000	28.66	1,000
Mixed	14.38	2,000	62.28	1,000
Table 8: Overview of few-shot data for NER experiments. Here, "Ave.  $|S|$ " corresponds to the average support set size of each domain. And "Sample" stands for the number of few-shot samples we build from each domain.
Experiment Data for Few-shot NER For named entity recognition, we utilize 4 different datasets: CoNLL-2003 (Sang and Meulder, 2003), GUM (Zeldes, 2017), WNUT-2017 (Derczynski et al., 2017) and Ontonotes (Pradhan et al., 2013), each of which contains data from only 1 domain. The 4 domains are News, Wiki, Social and Mixed. Detail of the original data set is showed in Table 7 and statistic of constructed few-shot data is showed in Table 8.
Model	1-shot Named Entity Recognition	Ave.
News	Wiki	Social	Mixed
Bi-LSMT	2.57 ±0.14	3.29 ±0.19	0.67 ±0.07	2.11 ±0.15	2.16 ±0.14
SimBERT	19.22 ±0.00	6.91 ±0.00	5.18 ±0.00	13.99 ±0.00	11.32 ±0.00
TransferBERT	4.75 ±1.42	0.57 ±0.32	2.71 ±0.72	3.46 ±0.54	2.87 ±0.75
MN	19.50 ±0.35	4.73 ±0.16	17.23 ±2.75	15.06 ±1.61	14.13 ±1.22
WPZ	3.64 ±0.08	2.00 ±0.02	0.92 ±0.04	0.66 ±0.03	1.80 ±0.04
WPZ+GloVe	9.40 ±0.06	3.23 ±0.01	2.29 ±0.02	2.56 ±0.01	4.37 ±0.03
WPZ+BERT	32.49 ±2.01	3.89 ±0.24	10.68 ±1.40	6.67 ±0.46	13.43 ±1.03
L-TapNet+CDT	44.30 ±3.15	12.04 ±0.65	20.80 ±1.06	15.17 ±1.25	23.08 ±1.53