SlowGuess/ACL_2020_OCR huggingface dataset:
ule is still able to further improve its performance with the knowledge produced by the toolkits even though the toolkits have worse-than-baseline results for the joint task. Second, among different types of knowledge, POS labels are the most effective ones that help the joint task. For instance, among BERT-based models, the one enhanced by POS knowledge from SCT achieves the best performance on most datasets, which is not surprising because such knowledge matches the outcome of the task. In addition, for BERT-based models enhanced by knowledge from BNP (i.e., BERT + POS (BNP) and BERT + Syn. (BNP)), syntactic constituents provide more improvement than POS labels on all CTB datasets. This observation could be explained by that BNP is originally designed for constituency parsing with CTB criteria; the syntactic constituents are complicated while effective when they are accurate. Third, while SCT and BNP were trained on CTB, whose tagset is very different from the two tagsets for UD, TwASP still outperforms the baselines on UD with the knowledge provided by SCT and BNP, indicating that syntactic knowledge is useful even when it uses different word segmentation and POS tagging criteria.
Table 4 shows the results of our best models (i.e. BERT and ZEN with POS (SCT)) and previous studies on the same datasets. Our approach
11We use the evaluation script from https://github. com/chakki-works/seqeval.
8291
CTB5	CTB6	CTB7	CTB9	UD1	UD2
Seg	Joint	Seg	Joint	Seg	Joint	Seg	Joint	Seg	Joint	Seg	Joint
Jiang et al. (2008)	97.85	93.41	-	-	-	-	-	-	-	-	-	-
Kruengkrai et al. (2009)	97.87	93.67	-	-	-	-	-	-	-	-	-	-
Sun (2011)	98.17	94.02	-	-	-	-	-	-	-	-	-	-
Wang et al. (2011)	98.11	94.18	95.79	91.12	95.65	90.46	-	-	-	-	-	-
Qian and Liu (2012)	97.85	93.53	-	-	-	-	-	-	-	-	-	-
Shen et al. (2014)	98.03	93.80	-	-	-	-	-	-	-	-	-	-
Kurita et al. (2017)	98.41	94.84	-	-	96.23	91.25	-	-	-	-	-	-
Shao et al. (2017)	98.02	94.38	-	-	-	-	96.67	92.34	95.16	89.75	95.09	89.42
Zhang et al. (2018)	98.50	94.95	96.36	92.51	96.25	91.87	-	-	-	-	-	-
BERT + POS (SCT)	98.77	96.77	97.43	94.82	97.31	94.12	97.75	94.87	98.32	95.60	98.33	95.46
ZEN + POS (SCT)	98.81	96.92	97.45	94.87	97.27	94.20	97.77	94.88	98.33	95.69	98.18	95.49
Table 4: Comparison (in F-scores of word segmentation and joint tagging) of TwASP (with BERT or ZEN encoder) with previous studies. Cells with “-” refer to the results are not reported or they are not applicable.
outperforms previous studies on the joint task and achieves new state-of-the-art performance on all datasets. While some of the previous studies use auto-analyzed knowledge (Wang et al., 2011; Zhang et al., 2018), they regard such knowledge as gold reference and consequently could suffer from errors in the auto-analyzed results. In contrast, our proposed model is able to selectively model the input information and to discriminate useful knowledge instances through the two-way attentions.
3.4 Cross-Domain Performance
Domain variance is an important factor affecting the performance of NLP systems (Guo et al., 2009; McClosky et al., 2010; Song and Xia, 2013). To further demonstrate the effectiveness of TwASP, we conduct cross-domain experiments on the eight genres of CTB9 using BERT and ZEN as the baseline and their enhanced version with POS knowledge from SCT. In doing so, we test on each genre with the models trained on the data from all other genres. The results on both segmentation and the joint task are reported in Table 5, where the SCT results are also included as a reference.
The comparison between the baselines and TwASP with POS knowledge clearly shows the consistency of performance improvement with two-way attentions, where for both BERT and ZEN, TwASP outperforms the baselines for all genres on the joint labels. In addition, similar to the observations from the previous experiment, both accurate and inaccurate POS knowledge are able to help the joint task. For example, although the SCT results on several genres (e.g., CS, DF, SC) are much worse than of the BERT baseline, the POS labels produced by SCT can still enhance TwASP on word segmentation and joint tagging with the proposed two-way attention module.
4.1 The Effect of Two Attention Ways
In the first analysis, we compare our two-way attention with normal attention. For normal attention, we experiment three ways of incorporating context features and knowledge: (1) using context features and knowledge together in the attention, where all features or knowledge instances are equally treated in it; (2) using context features only; and (3) using knowledge only. We run these experiments with BERT encoder and POS knowledge from SCT on CTB5 and report the results in Table 6. Overall, the two-way attentions outperform all three settings for normal attention, which clearly indicates the validity of using two attention ways for features and knowledge, i.e., compared to (1), as well as the advantage of learning from both of them, i.e., compared to (2) and (3). Interestingly, in the three settings, (3) outperforms (1), which could be explained by that, with normal attention, mixed feature and knowledge instances in it may make it difficult to weight them for the joint task.
There are other methods for using both context features and knowledge in a neural framework, such as key-value memory networks (kvMN) (Miller et al., 2016), which is demonstrated to improve CWS by Tian et al. (2020). Thus we compare our approach with kvMN, in which context features are mapped to keys and knowledge to values. We follow the standard protocol of the kvMN, e.g., addressing keys by  $S_{i}$  and reading values from  $\mathcal{K}_i$  through the corresponding knowledge for each key, computing weights from all key embeddings, and outputting the weighted embeddings from all values. The result from the kvMN is reported at the last row of Table 6, where its performance is not as good as the two-way attentions, and even
8292
Genre	SCT	BERT	BERT+POS	ZEN	ZEN+POS
Seg	Joint	Seg	Joint	Seg	Joint	Seg	Joint	Seg	Joint
BC	96.27	93.55	96.29	92.08	96.38	92.34	96.48	92.25	96.63	92.41
BN	96.98	93.98	96.93	93.73	97.20	94.02	97.05	93.91	97.21	94.14
CS	89.83	81.93	95.17	89.18	95.14	89.46	95.10	89.24	95.87	89.67
DF	91.34	84.28	96.79	92.02	96.44	92.44	96.33	92.11	96.55	92.51
MZ	95.69	91.99	95.62	91.97	95.83	92.17	95.69	92.00	95.78	92.18
NW	97.41	94.75	97.55	94.44	97.49	94.64	97.49	94.51	97.57	94.70
SC	84.87	76.55	95.97	91.13	96.27	91.77	96.09	91.47	96.38	91.85
WB	95.99	92.86	95.09	89.59	95.11	89.85	95.10	89.74	95.35	90.10
Table 5: Experimental results (the F-scores for word segmentation and joint tagging) from baselines and TwASP with different encoders on eight genres of CTB9. The incorporated knowledge is the POS labels from SCT.
Ways	Seg	Joint
Feature	Knowledge	F	ROOV	F
✓	✓	98.55	87.28	96.62
✓	×	98.67	87.38	96.50
×	✓	98.71	88.17	96.69
Two-way Attentions	98.77	88.13	96.77
Key-value Memory	98.62	88.51	96.58
worse than using normal attention with setting (3). The reason could be straightforward: the output of kvMN is built upon value (knowledge) embeddings and therefore information from key (context feature) embeddings does not directly contribute to it other than providing weights for the value. As a result, kvMN acts in a similar yet inferior $^{12}$  way of setting (3) where only knowledge is used.
4.2 Knowledge Ensemble
Since every type of knowledge works well in our model, it is expected to investigate how the model performs when multiple types of knowledge are used together. To this end, we run experiments on CTB5 to test on our BERT-based TwASP with knowledge ensemble, where two ensemble strategies, i.e., averaging and concatenation, are applied with respect to how  $\mathbf{a}_i$  for each knowledge type is combined with others. The results are reported in Table 7. In this table, the first seven rows (ID: 1-7) indicate that different types of knowledge are
Table 6: Performance comparison among different ways of knowledge integration, including normal attention (with respect to what knowledge type is used), the two-way attention, and key-value memory networks.
ID	SCT	BNP	Joint F
POS	Syn.	Dep.	POS	Syn.	Σ	⊕
1	✓	✓	96.79	96.80
2	✓	✓	96.78	96.81
3	✓	✓	96.79	96.80
4	✓	✓	✓	96.82	96.87
5	✓	✓	96.76	96.81
6	✓	✓	96.81	96.83
7	✓	✓	✓	96.82	96.84
8	✓	✓	✓	✓	✓	96.87	96.90
Table 7: Comparison of different knowledge ensemble results, which are presented by the joint tagging  $F$ -scores from our BERT-based TwASP on CTB5.  $\sum$  and  $\oplus$  refer to averaging and concatenation of attentions from different knowledge types, respectively. As a reference, the best result on CTB5 for BERT-based model without knowledge ensemble is  $96.77\%$  achieved by BERT + POS (SCT) (see Table 3).
combined according to whether they come from the same toolkit (ID: 1-5) or belong to the same category (ID: 6 and 7); and the last row (ID: 8) is for the case that all types of knowledge are combined.
There are several observations. First, compared to only using one type of knowledge (refer to Table 3), knowledge ensemble improves model performance where more knowledge types contribute to better results. The best model is thus obtained when all knowledge (from each toolkit and from both toolkits) are used. Second, knowledge in the same type from different toolkits may complement to each other and thus enhance model performance accordingly, which is confirmed by the results from the models assembling POS (or Syn+Dep) information from both SCT and BNP. Third, for different ensemble strategies, concatenation tends to perform better than averaging, which is not surprising since concatenation actually turns the model into a multi-way structure for knowledge integration.
<sup>12</sup>The "inferior" is explained by that, in kvMN, the value weights are inaccurate because they are computed with respect to the contribution of keys rather than knowledge instances.
8293
Figure 4: Comparison of joint tagging results between BERT and BERT+Dep (SCT) on an example sentence.
When the toolkit provides accurate knowledge, it is not surprising that our two-way attention model would benefit from the auto-analyzed knowledge. Interestingly, even when the toolkit provides inaccurate output, our model might still be able to benefit from such output. Figure 4 shows such an example, where our system uses BERT+Dep using SCT and the baseline system is BERT without two-way attention. The sentence contains an ambiguity character bigram "马上", which has two possible interpretations, "马上\_AD" (immediately) and "马\_NN/上\_LC" (on the horse). The second one is correct, yet the baseline tagger chooses the former because "马上" (immediately) is a very common adverb. Although SCT also chooses the wrong segmentation and thus has an incorrect dependency structure, our system is still able to produce correct segmentation and POS tags. One plausible explanation for this is that the inaccurate dependency structure includes an advmod link between "马上" (immediately) and "很好"(very good). Because such a dependency pair seldom appears in the corpus, the attention from such knowledge is weak and hence encourages our system to choose the correct word segmentation and POS tags.
There are basically two approaches to CWS and POS tagging: to perform POS tagging right after word segmentation in a pipeline, or to conduct the two tasks simultaneously, known as joint CWS and POS tagging. In the past two decades, many studies have shown that joint tagging outperforms the pipeline approach (Ng and Low, 2004; Jiang et al., 2008, 2009; Wang et al., 2011; Sun, 2011; Zeng et al., 2013). In recent years, neural methods started to play a dominant role for this task (Zheng et al., 2013; Kurita et al., 2017; Shao et al., 2017; Zhang et al., 2018), where some of them tried to incorporate extra knowledge in their studies. For
example, Kurita et al. (2017) exploited to model n-grams to improve the task; Shao et al. (2017) extended the idea by incorporating pre-trained n-gram embeddings, as well as radical embeddings, into character representations. Zhang et al. (2018) tried to leverage the knowledge from character embeddings, trained on an automatically tagged corpus by a baseline tagger. Compared to these previous studies, TwASP provides a simple, yet effective, neural model for joint tagging, without requiring a complicated mechanism of incorporating different features or pre-processing a corpus.
In this paper, we propose neural approach with a two-way attention mechanism to incorporate autoanalyzed knowledge for joint CWS and POS tagging, following a character-based sequence labeling paradigm. Our proposed attention module learns and weights context features and their corresponding knowledge instances in two separate ways, and use the combined attentions from the two ways to enhance the joint tagging. Experimental results on five benchmark datasets illustrate the validity and effectiveness of our model, where the two-way attentions can be integrated with different encoders and provide consistent improvements over baseline taggers. Our model achieves state-of-the-art performance on all the datasets. Overall, this work presents an elegant way to use autoanalyzed knowledge and enhance neural models with existing NLP tools. For future work, we plan to apply the same methodology to other NLP tasks.
Xiang Ao was partially supported by the National Natural Science Foundation of China under Grant No. 61976204, U1811461, the Natural Science Foundation of Chongqing under Grant No. cstc2019jcyj-msxmX0149 and the Project of Youth Innovation Promotion Association CAS.
Wenliang Chen, Yujie Zhang, and Hitoshi Isahara. 2006. An Empirical Study of Chinese Chunking. In Proceedings of the COLING/ACL 2006 Main Conference Poster Sessions, pages 97-104, Sydney, Australia.
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of
8294
Deep Bidirectional Transformers for Language Understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 4171-4186.
Shizhe Diao, Jiaxin Bai, Yan Song, Tong Zhang, and Yonggang Wang. 2019. ZEN: Pre-training Chinese Text Encoder Enhanced by N-gram Representations. ArXiv, abs/1911.00720.
Edouard Grave, Piotr Bojanowski, Prakhar Gupta, Armand Joulin, and Tomas Mikolov. 2018. Learning Word Vectors for 157 Languages. In Proceedings of the Eleventh International Conference on Language Resources and Evaluation (LREC 2018), Miyazaki, Japan.
Honglei Guo, Huijia Zhu, Zhili Guo, Xiaoxun Zhang, Xian Wu, and Zhong Su. 2009. Domain Adaptation with Latent Semantic Association for Named Entity Recognition. In Proceedings of Human Language Technologies: The 2009 Annual Conference of the North American Chapter of the Association for Computational Linguistics, pages 281-289, Boulder, Colorado.
Binxuan Huang and Kathleen M Carley. 2019. Syntax-Aware Aspect Level Sentiment Classification with Graph Attention Networks. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 5472-5480.
Zhongqiang Huang, Mary Harper, and Wen Wang. 2007. Mandarin Part-of-Speech Tagging and Discriminative Reranking. In Proceedings of the 2007 Joint Conference on Empirical Methods in Natural Language Processing and Computational Natural Language Learning (EMNLP-CoNLL), pages 1093-1102, Prague, Czech Republic.
Wenbin Jiang, Liang Huang, and Qun Liu. 2009. Automatic Adaptation of Annotation Standards: Chinese Word Segmentation and POS Tagging – A Case Study. In Proceedings of the Joint Conference of the 47th Annual Meeting of the ACL and the 4th International Joint Conference on Natural Language Processing of the AFNLP, pages 522–530, Suntec, Singapore.
Wenbin Jiang, Liang Huang, Qun Liu, and Yajuan Lü. 2008. A Cascaded Linear Model for Joint Chinese Word Segmentation and Part-of-Speech Tagging. In Proceedings of ACL-08: HLT, pages 897–904, Columbus, Ohio.
Nikita Kitaev and Dan Klein. 2018. Constituency Parsing with a Self-Attentive Encoder. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 2676–2686, Melbourne, Australia.
Canasai Kruengkrai, Kiyotaka Uchimoto, Jun'ichi Kazama, Yiou Wang, Kentaro Torisawa, and Hitoshi Isahara. 2009. An Error-Driven Word-Character Hybrid Model for Joint Chinese Word Segmentation and POS Tagging. In Proceedings of the Joint Conference of the 47th Annual Meeting of the ACL and the 4th International Joint Conference on Natural Language Processing of the AFNLP, pages 513-521, Suntec, Singapore.
Abhishek Kumar, Daisuke Kawahara, and Sadao Kurohashi. 2018. Knowledge-Enriched Two-Layered Attention Network for Sentiment Analysis. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 2 (Short Papers), pages 253–258, New Orleans, Louisiana.
Shuhei Kurita, Daisuke Kawahara, and Sadao Kurohashi. 2017. Neural Joint Model for Transition-based Chinese Syntactic Analysis. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1204-1214, Vancouver, Canada.
Christopher Manning, Mihai Surdeanu, John Bauer, Jenny Finkel, Steven Bethard, and David McClosky. 2014. The Stanford CoreNLP Natural Language Processing Toolkit. In Proceedings of 52nd Annual Meeting of the Association for Computational Linguistics: System Demonstrations, pages 55-60, Baltimore, Maryland.
Katerina Margatina, Christos Baziotis, and Alexandros Potamianos. 2019. Attention-based Conditioning Methods for External Knowledge Integration. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 3944-3951, Florence, Italy.
David McClosky, Eugene Charniak, and Mark Johnson. 2010. Automatic Domain Adaptation for Parsing. In Human Language Technologies: The 2010 Annual Conference of the North American Chapter of the Association for Computational Linguistics, pages 28-36, Los Angeles, California.
Alexander Miller, Adam Fisch, Jesse Dodge, Amir-Hossein Karimi, Antoine Bordes, and Jason Weston. 2016. Key-Value Memory Networks for Directly Reading Documents. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pages 1400-1409.
Hwee Tou Ng and Jin Kiat Low. 2004. Chinese Part-of-Speech Tagging: One-at-a-Time or All-at-Once? Word-Based or Character-Based? In Proceedings of the 2004 Conference on Empirical Methods in Natural Language Processing, pages 277-284, Barcelona, Spain.
Joakim Nivre, Marie-Catherine De Marneffe, Filip Ginter, Yoav Goldberg, Jan Hajic, Christopher D Manning, Ryan McDonald, Slav Petrov, Sampo Pyysalo,
Natalia Silveira, et al. 2016. Universal Dependencies v1: A Multilingual Treebank Collection. In Proceedings of the Tenth International Conference on Language Resources and Evaluation (LREC 2016), pages 1659-1666.
Xian Qian and Yang Liu. 2012. Joint Chinese Word Segmentation, POS Tagging and Parsing. In Proceedings of the 2012 Joint Conference on Empirical Methods in Natural Language Processing and Computational Natural Language Learning, pages 501-511, Jeju Island, Korea.
Colin Raffel and Daniel PW Ellis. 2015. Feed-Forward Networks with Attention Can Solve Some Long-Term Memory Problems. arXiv preprint arXiv:1512.08756.
Dominic Seyler, Tatiana Dembelova, Luciano Del Corro, Johannes Hoffart, and Gerhard Weikum. 2018. A Study of the Importance of External Knowledge in the Named Entity Recognition Task. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 241-246.
Yan Shao, Christian Hardmeier, Jörg Tiedemann, and Joakim Nivre. 2017. Character-based Joint Segmentation and POS Tagging for Chinese using Bidirectional RNN-CRF. In Proceedings of the Eighth International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 173-183, Taipei, Taiwan.
Mo Shen, Hongxiao Liu, Daisuke Kawahara, and Sadao Kurohashi. 2014. Chinese Morphological Analysis with Character-level POS Tagging. In Proceedings of the 52nd Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 253-258, Baltimore, Maryland.
Yan Song, Chia-Jung Lee, and Fei Xia. 2017. Learning Word Representations with Regularization from Prior Knowledge. In Proceedings of the 21st Conference on Computational Natural Language Learning (CoNLL 2017), pages 143-152, Vancouver, Canada.
Yan Song, Shuming Shi, Jing Li, and Haisong Zhang. 2018. Directional Skip-Gram: Explicitly Distinguishing Left and Right Context for Word Embeddings. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 2, pages 175–180, New Orleans, Louisiana.
Yan Song and Fei Xia. 2013. A Common Case of Jekyll and Hyde: The Synergistic Effect of Using Divided Source Training Data for Feature Augmentation. In Proceedings of the Sixth International Joint Conference on Natural Language Processing, pages 623-631, Nagoya, Japan.
Weiwei Sun. 2011. A Stacked Sub-Word Model for Joint Chinese Word Segmentation and Part-of-Speech Tagging. In Proceedings of the 49th Annual Meeting of the Association for Computational
Linguistics: Human Language Technologies, pages 1385-1394, Portland, Oregon, USA.
Yuanhe Tian, Yan Song, Fei Xia, Tong Zhang, and Yonggang Wang. 2020. Improving Chinese Word Segmentation with Wordhood Memory Networks. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, Seattle, Washington, USA.
Yiou Wang, Jun'ichi Kazama, Yoshimasa Tsuruoka, Wenliang Chen, Yujie Zhang, and Kentaro Torisawa. 2011. Improving Chinese Word Segmentation and POS Tagging with Semi-supervised Methods Using Large Auto-Analyzed Data. In Proceedings of 5th International Joint Conference on Natural Language Processing, pages 309-317, Chiang Mai, Thailand.
Naiwen Xue, Fei Xia, Fu-Dong Chiou, and Marta Palmer. 2005. The Penn Chinese TreeBank: Phrase structure annotation of a large corpus. Natural language engineering, 11(2):207-238.
Ikuya Yamada, Akari Asai, Jin Sakuma, Hiroyuki Shindo, Hideaki Takeda, Yoshiyasu Takefuji, and Yuji Matsumoto. 2020. Wikipedia2Vec: An Efficient Toolkit for Learning and Visualizing the Embeddings of Words and Entities from Wikipedia. arXiv preprint 1812.06280v3.
Xiaodong Zeng, Derek F. Wong, Lidia S. Chao, and Isabel Trancoso. 2013. Graph-based Semi-Supervised Model for Joint Chinese Word Segmentation and Part-of-Speech Tagging. In Proceedings of the 51st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 770-779, Sofia, Bulgaria.
Hongming Zhang, Jiaxin Bai, Yan Song, Kun Xu, Changlong Yu, Yangqiu Song, Wilfred Ng, and Dong Yu. 2019. Multiplex Word Embeddings for Selectional Preference Acquisition. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 5247-5256, Hong Kong, China.
Meishan Zhang, Nan Yu, and Guohong Fu. 2018. A Simple and Effective Neural Model for Joint Word Segmentation and POS Tagging. IEEE/ACM Transactions on Audio, Speech and Language Processing (TASLP), 26(9):1528-1538.
Xiaoqing Zheng, Hanyang Chen, and Tianyu Xu. 2013. Deep Learning for Chinese Word Segmentation and POS Tagging. In Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing, pages 647-657, Seattle, Washington, USA.
8296