SlowGuess/ACL_2020_OCR huggingface dataset:
There are several observations drawn from the results. First, the overall comparison clearly indicates that, WMSEG (i.e., the model with wordhood memories) outperforms the baseline (i.e., the model without wordhood memories) for all 30 pairs in terms of F-scores and for 25 pairs in terms of  $R_{OOV}$ . Second, the proposed memory module works smoothly with different encoders and decoders, where some improvement is pretty signifi
cant; for instance, when using Bi-LSTM as the encoder and CRF as the decoder, WMSEG improves the F-score on the AS dataset from 94.39 to 95.07 and  $R_{OOV}$  from 61.59 to 68.17. With BERT or ZEN as the encoder, even when the baseline system performs very well, the improvement of WMSEG on F-scores is still decent. Third, among the models with ZEN, the ones with the memory module further improve their baselines, although the context information carried by n-grams is already learned in pre-training ZEN. This indicates that wordhood information provides additional cues (besides the contextual features) that can benefit CWS, and our proposed memory module is able to provide further task-specific guidance to an n-gram integrated encoder. Fourth, the wordhood memory shows its robustness with different lexicon size when we consider WMSEG's performance with the lexicon statistics reported in Table 4 together. To summarize, the results in this experiment not only confirm that wordhood information is a simple yet effective source of knowledge to help CWS without requiring external support such as a well-defined dictionary or manually crafted heuristics, but also fully illustrate that the design of our model can effectively integrate this type of knowledge.
To further illustrate the validity and the effectiveness of WMSEG, we compare our best-performing model with the ones in previous studies on the same benchmark datasets. The comparison is presented in Table 7, where WMSEG (both the one with BERT and ZEN) outperforms all existing models with respect to the F-scores and achieves new state-of-the-art performance on all datasets.
MSR	PKU	AS	CityU	CTB6
F	ROOV	F	ROOV	F	ROOV	F	ROOV	F	ROOV
ZHANG ET AL. (2013)	97.5	-	96.1	73.1	-	-	-	-	-	-
PEI ET AL. (2014)	97.2	-	95.2	-	-	-	-	-	-	-
MA AND HINRICHs (2015)	96.6	87.2	95.1	76.0	-	-	-	-	-	-
CHEN ET AL. (2015)	97.4	-	96.5	-	-	-	-	-	96.0	-
XU AND SUN (2016)	96.3	-	96.1	-	-	-	-	-	95.8	-
ZHANG ET AL. (2016)	97.7	-	95.7	-	-	-	-	-	95.95	-
CHEN ET AL. (2017)	96.04	71.60	94.32	72.64	94.75	75.34	95.55	81.40	-	-
WANG AND XU (2017)	98.0	-	96.5	-	-	-	-	-	-	-
ZHOU ET AL. (2017)	97.8	-	96.0	-	-	-	-	-	96.2	-
MA ET AL. (2018)	98.1	80.0	96.1	78.8	96.2	70.7	97.2	87.5	96.7	85.4
GONG ET AL. (2019)	97.78	64.20	96.15	69.88	95.22	77.33	96.22	73.58	-	-
HIGASHIYAMA ET AL. (2019)	97.8	-	-	-	-	-	-	-	96.4	-
QIU ET AL. (2019)	98.05	78.92	96.41	78.91	96.44	76.39	96.91	86.91	-	-
WMSEG (BERT-CRF)	98.28	86.67	96.51	86.76	96.58	78.48	97.80	87.57	97.16	88.00
WMSEG (ZEN-CRF)	98.40	84.87	96.53	85.36	96.62	79.64	97.93	90.15	97.25	88.46
Table 7: Performance (F-score) comparison between WMSEG (BT-CRF and ZEN-CRF with woodhood memory networks) and previous state-of-the-art models on the test set of five benchmark datasets.
4.2 Cross-Domain Performance
As domain variance is always an important factor affecting the performance of NLP systems especially word semgenters (Song et al., 2012; Song and Xia, 2013), in addition to the experiments on benchmark datasets, we also run WMSEG on CTB7 across domains (genres in this case) with and without the memory module. To test on each genre, we use the union of the data from the other four genres to train our segmenter and use AV to extract n-grams from the entire raw text from CTB7 in this experiment. Table 8 reports the results in F-score and OOV recall, which show a similar trend as that in Table 6, where WMSEG outperforms baselines for all five genres. Particularly, for genres with large domain variance (e.g., the ones with high OOV rates such as MZ and WEB), CWS is difficult, and its relatively low F-scores in Table 8 from baseline models confirm that. Yet WMSEG offers a decent way to improve cross-domain CWS performance without any help from external knowledge or complicated model design, which further illustrates the effectiveness of the memory module. The reason could be that many n-grams are shared in both training and test data; these n-grams with their wordhood information present a strong indication to the model on what combinations of characters can be treated as words, even though some of them never appear in the training data.
4.3 Effect of Using Different  $\mathcal{N}$
To analyze the robustness of WMSEG with respect to the lexicon, we compare four ways (ID: 2-5 in Table 9) of constructing the lexicon  $(\mathcal{N})$ : the first one
simply uses the vocabulary from the training data (marked as GOLD LABEL in Table 9; ID: 2); the other three ways use AV to extract n-grams from the unsegmented training data only (ID: 3), the test data only (ID: 4), and training + test set (ID: 5), respectively.7 Table 9 shows the results of running BERT-CRF on the WEB genre of CTB7 without the wordhood memories (ID: 1) and with the memories (ID: 2-5), following the cross-domain setting in  $\S 4.2$ . While the four methods with memories achieve similar results on the  $F$  score, indicating the robustness of our proposed framework, the one that builds  $\mathcal{N}$  using the raw texts from both training and test sets through unsupervised method (ID: 5) achieves the biggest improvement on  $R_{OOV}$ , demonstrating the advantage of including the unlabeled test set by incorporating the results from unsupervised wordhood measures into the models.
4.4 Effect of Different Wordhood Measures
WMSEG provides a general way of integrating wordhood information for CWS, we expect other wordhood measures to play the same role in it. Therefore, we test PMI and DLG in our model and compare them with the previous results from AV (see Table 6). Specifically, we use our best performing BERT-based model, i.e., BERT-CRF, with the n-gram lexicons constructed by the aforementioned three measures and run it on all benchmark datasets. We draw the histograms of the F-scores obtained from WMSEG with each measure (red, green, and blue bars for AV, PMI, and DLG, re
One could also use an external corpus to build  $N$ , which is not considered in this experiment.
CONFIG	BC	BN	MZ	NW	WEB
EN-DN	WM	F	ROOV	F	ROOV	F	ROOV	F	ROOV	F	ROOV
BL-SM	×	93.73	63.39	93.65	68.88	90.55	66.95	93.70	69.57	90.81	55.50
✓	94.04	63.53	93.91	72.32	90.76	65.65	93.83	72.40	91.22	56.62
BL-CRF	×	93.95	65.60	93.87	71.89	90.67	67.13	93.87	72.17	91.12	57.51
✓	94.21	66.81	94.11	74.22	90.95	67.29	93.96	74.38	91.49	58.37
BT-SM	×	96.27	80.76	96.88	87.90	94.97	84.45	97.08	89.78	94.82	74.00
✓	96.41	81.15	97.00	89.47	95.10	85.48	97.24	91.96	95.00	75.51
BT-CRF	×	96.25	79.04	96.87	89.15	94.94	85.27	96.99	91.34	94.79	75.58
✓	96.43	81.29	97.09	90.29	95.11	85.32	97.21	92.48	95.03	76.30
ZEN-SM	×	96.39	79.97	96.95	88.93	95.05	85.14	97.17	91.33	94.03	75.33
✓	96.45	81.34	97.03	89.78	95.06	85.60	97.21	91.73	95.08	75.60
ZEN-CRF	×	96.30	80.05	96.97	90.38	94.93	85.64	97.10	91.03	94.90	74.98
✓	96.50	80.44	97.11	90.29	95.13	85.96	97.24	91.68	95.04	75.74
Table 8: Experimental results on five genres of CTB7. Abbreviations follow the same notation in Table 6.
ID	TRAIN	TEST	GOLD	LABEL	F	RooV
1	-	-	-	94.79	75.58
2	×	×	√	+0.22	+0.21
3	√	×	×	+0.21	+0.20
4	×	√	×	+0.23	+0.33
5	√	√	×	+0.24	+0.72
Table 9: Comparisons of performance gain on the WEB genre of CTB7 with respect to the baseline BERT-CRF model when the n-gram lexicon  $\mathcal{N}$  for WMSEG is built upon different sources.  $\sqrt{}$  and  $\times$  refer to if a corresponding data source is used or not, respectively.
spectively) in Figure 2, where the F-scores of the baseline model are also presented in orange bars.
As shown in the figure, the performances of using the three measures are very similar, which indicates that WMSEG is able to robustly incorporate the wordhood information from various measures, despite that those measures focus on different aspects of n-grams when determining whether the n-grams should be treated as words. Particularly, consider that the lexicons produced by the three measures are rather different in their sizes (as shown in Table 4), the results in Figure 2 strongly demonstrate the effectiveness of our proposed approach in learning with a limited number of n-grams. This observation also reveals the possibility that many n-grams may be redundant for our model, and WMSEG is thus able to identify the most useful ones from them, which is analyzed in the case study.
To investigate how the memory learns from the wordhood information carried by n-grams, we conduct a case study with an example input sentence "他/从小/学/电脑/技术" (He learned computer techniques since childhood). In this sentence, the
Figure 2: The F-scores of WMSEG (BERT) using three different wordhood measures, namely AV (red), PMI (green), and DLG (blue), on five benchmark datasets.
n-gram “从小学” is ambiguous with two possible interpretations: “从小/学” (learn since childhood) and “从/小学” (from primary school). Native Chinese speakers can easily choose the first one with the given context but a word segmenter might incorrectly choose the second segmentation.
We feed this case into our BERT-CRF model with the memory module. In Figure 3, we visualize the resulted weights that learned from keys (a) and values (b) of the memory, as well as from the final tagger (c). The heatmaps of all keys and values in the memory with respect to each corresponding input character clearly illustrate that the appropriate n-grams, e.g., "他" (he), "学" (learn), "从小" (from childhood), etc., receive higher weights than others and the corresponding values for them are also emphasized, which further affects final CWS tagging so that the weight distributions from (b) and (c) look alike to each other. Therefore, this visualization explains, to some extent, that the proposed memory mechanism can identify and distinguish important n-grams within a certain context and thus improves CWS performance accordingly.
Figure 3: Heatmaps of weights learned for (a) keys and (b) values in the memory, and (c) the tags from the decoder, with respect to each character in an input sentence. Higher weights are visualized with darker colors.
As one of the most fundamental NLP tasks for Chinese language processing, CWS has been studied for decades, with two steams of methods, i.e., word-based and character-based ones (Xue and Shen, 2003; Peng et al., 2004; Levow, 2006; Zhao et al., 2006; Zhao and Kit, 2008; Li and Sun, 2009; Song et al., 2009a; Li, 2011; Sun and Xu, 2011; Mansur et al., 2013; Zhang et al., 2013; Pei et al., 2014; Chen et al., 2015; Ma and Hinrichs, 2015; Liu et al., 2016; Zhang et al., 2016; Wang and Xu, 2017; Zhou et al., 2017; Chen et al., 2017; Ma et al., 2018; Higashiyama et al., 2019; Gong et al., 2019; Qiu et al., 2019). Among these studies, most of them follow the character-based paradigm to predict segmentation labels for each character in an input sentence; n-grams are used in some of these studies to enhance model performance, which is also observed in many other NLP tasks (Song et al., 2009b; Xiong et al., 2011; Shrestha, 2014; Shi et al., 2016; Diao et al., 2019). Recently, CWS benefits from neural networks and further progress are made with embeddings (Pei et al., 2014; Ma and Hinrichs, 2015; Liu et al., 2016; Zhang et al., 2016; Wang and Xu, 2017; Zhou et al., 2017), recurrent neural models (Chen et al., 2015; Ma et al., 2018; Higashiyama et al., 2019; Gong et al., 2019) and even adversarial learning (Chen et al., 2017). To enhance CWS with neural models, there were studies leverage external information, such as vocabularies from auto-segmented external corpus (Wang and Xu, 2017; Higashiyama et al., 2019), where Higashiyama et al. (2019) introduced a word attention mechanism to learn from large granular texts during the CWS process. In addition, the studies from Chen et al. (2017) and Qiu et al. (2019) try to improve CWS by learning from data annotated through different segmentation criteria. Moreover, there is a study leveraging auto-analyzed syntactic
knowledge obtained from off-the-shelf toolkits to help CWS and part-of-speech tagging (Tian et al., 2020). Compare to these studies, WMSEG offers an alternative solution to robustly enhancing neural CWS models without requiring external resources.
In this paper, we propose WMSEG, a neural framework for CWS using wordhood memory networks, which maps n-grams and their wordhood information to keys and values in it and appropriately models the values according to the importance of keys in a specific context. The framework follows the sequence labeling paradigm, and the encoders and decoders in it can be implemented by various prevailing models. To the best of our knowledge, this is the first work using key-value memory networks and utilizing wordhood information for neural models in CWS. Experimental results on various widely used benchmark datasets illustrate the effectiveness of WMSEG, where state-of-the-art performance is achieved on all datasets. Further experiments and analyses also demonstrate the robustness of WMSEG in the cross-domain scenario as well as when using different lexicons and wordhood measures.
Xinchi Chen, Xipeng Qiu, Chenxi Zhu, Pengfei Liu, and Xuanjing Huang. 2015. Long Short-Term Memory Neural Networks for Chinese Word Segmentation. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, pages 1197-1206.
Xinchi Chen, Zhan Shi, Xipeng Qiu, and Xuanjing Huang. 2017. Adversarial Multi-Criteria Learning for Chinese Word Segmentation. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1193-1203.
Alexis Conneau, Douwe Kiela, Holger Schwenk, Loic Barrault, and Antoine Bordes. 2017. Supervised Learning of Universal Sentence Representations from Natural Language Inference Data. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pages 670-680, Copenhagen, Denmark.
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 4171-4186.
Shizhe Diao, Jiaxin Bai, Yan Song, Tong Zhang, and Yonggang Wang. 2019. ZEN: Pre-training Chinese Text Encoder Enhanced by N-gram Representations. ArXiv, abs/1911.00720.
Thomas Emerson. 2005. The Second International Chinese Word Segmentation Bakeoff. In Proceedings of the fourth SIGHAN workshop on Chinese language Processing, pages 123-133.
Haodi Feng, Kang Chen, Xiaotie Deng, and Weimin Zheng. 2004. Accessor Variety Criteria for Chinese Word Extraction. Computational Linguistics, 30(1):75-93.
Jingjing Gong, Xinchi Chen, Tao Gui, and Xipeng Qiu. 2019. Switch-LSTMs for Multi-Criteria Chinese Word Segmentation. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 33, pages 6457-6464.
Shohei Higashiyama, Masao Utiyama, Eiichiro Sumita, Masao Ideuchi, Yoshiaki Oida, Yohei Sakamoto, and Isaac Okada. 2019. Incorporating Word Attention into Character-Based Word Segmentation. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 2699-2709.
Chunyu Kit and Yorick Wilks. 1999. Unsupervised Learning of Word Boundary with Description Length Gain. In EACL 1999: CoNLL-99 Computational Natural Language Learning, pages 1-6.
Gina-Anne Levow. 2006. The Third International Chinese Language Processing Bakeoff: Word Segmentation and Named Entity Recognition. In Proceedings of the Fifth SIGHAN Workshop on Chinese Language Processing, pages 108-117.
Zhongguo Li. 2011. Parsing the Internal Structure of Words: A New Paradigm for Chinese Word Segmentation. In Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies, pages 1405-1414, Portland, Oregon, USA.
Zhongguo Li and Maosong Sun. 2009. Punctuation as Implicit Annotations for Chinese Word Segmentation. Computational Linguistics, 35(4):505-512.
Yijia Liu, Wanxiang Che, Jiang Guo, Bing Qin, and Ting Liu. 2016. Exploring Segment Representations for Neural Segmentation Models. arXiv preprint arXiv:1604.05499.
Ziqing Liu, Enwei Peng, Shixing Yan, Guozheng Li, and Tianyong Hao. 2018. T-Know: a Knowledge Graph-based Question Answering and Information Retrieval System for Traditional Chinese Medicine. In Proceedings of the 27th International Conference on Computational Linguistics: System Demonstrations, pages 15-19, Santa Fe, New Mexico.
Ji Ma, Kuzman Ganchev, and David Weiss. 2018. State-of-the-art Chinese Word Segmentation with Bi-LSTMs. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 4902-4908.
Jianqiang Ma and Erhard Hinrichs. 2015. Accurate Linear-Time Chinese Word Segmentation via Embedding Matching. In Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 1733-1743.
Mairgup Mansur, Wenzhe Pei, and Baobao Chang. 2013. Feature-based Neural Language Model and Chinese Word Segmentation. In Proceedings of the Sixth International Joint Conference on Natural Language Processing, pages 1271-1277, Nagoya, Japan.
Alexander Miller, Adam Fisch, Jesse Dodge, Amir-Hossein Karimi, Antoine Bordes, and Jason Weston. 2016. Key-Value Memory Networks for Directly Reading Documents. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pages 1400-1409.
Wenzhe Pei, Tao Ge, and Baobao Chang. 2014. Maxmargin Tensor Neural Network for Chinese Word Segmentation. In Proceedings of the 52nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 293-303.
Fuchun Peng, Fangfang Feng, and Andrew McCallum. 2004. Chinese Segmentation and New Word Detection Using Conditional Random Fields. In Proceedings of the 20th international conference on Computational Linguistics, page 562.
Xipeng Qiu, Hengzhi Pei, Hang Yan, and Xuanjing Huang. 2019. Multi-Criteria Chinese Word Segmentation with Transformer. arXiv preprint arXiv:1906.12035.
Yangyang Shi, Kaisheng Yao, Le Tian, and Daxin Jiang. 2016. Deep LSTM based Feature Mapping for Query Classification. In Proceedings of the 2016 Conference of the North American Chapter of
the Association for Computational Linguistics: Human Language Technologies, pages 1501-1511, San Diego, California.
Prajwol Shrestha. 2014. Incremental N-gram Approach for Language Identification in Code-Switched Text. In Proceedings of the First Workshop on Computational Approaches to Code Switching, pages 133-138, Doha, Qatar.
Damien Sileo, Tim Van De Cruys, Camille Pradel, and Philippe Muller. 2019. Mining Discourse Markers for Unsupervised Sentence Representation Learning. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 3477-3486, Minneapolis, Minnesota.
Yan Song, Dongfeng Cai, Guiping Zhang, and Hai Zhao. 2009a. Approach to Chinese Word Segmentation Based on Character-word Joint Decoding. Journal of Software, 20(9):2236-2376.
Yan Song, Jiaqing Guo, and Dongfeng Cai. 2006. Chinese Word Segmentation Based on an Approach of Maximum Entropy Modeling. In Proceedings of the Fifth SIGHAN Workshop on Chinese Language Processing, pages 201-204, Sydney, Australia.
Yan Song, Chunyu Kit, and Xiao Chen. 2009b. Transliteration of Name Entity via Improved Statistical Translation on Character Sequences. In Proceedings of the 2009 Named Entities Workshop: Shared Task on Transliteration (NEWS 2009), pages 57-60, Suntec, Singapore.
Yan Song, Prescott Klassen, Fei Xia, and Chunyu Kit. 2012. Entropy-based Training Data Selection for Domain Adaptation. In Proceedings of COLING 2012: Posters, pages 1191-1200, Mumbai, India.
Yan Song, Chia-Jung Lee, and Fei Xia. 2017. Learning Word Representations with Regularization from Prior Knowledge. In Proceedings of the 21st Conference on Computational Natural Language Learning (CoNLL 2017), pages 143-152, Vancouver, Canada.
Yan Song, Shuming Shi, Jing Li, and Haisong Zhang. 2018. Directional Skip-Gram: Explicitly Distinguishing Left and Right Context for Word Embeddings. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 2, pages 175–180, New Orleans, Louisiana.
Yan Song and Fei Xia. 2012. Using a Goodness Measurement for Domain Adaptation: A Case Study on Chinese Word Segmentation. In LREC, pages 3853-3860.
Yan Song and Fei Xia. 2013. A Common Case of Jekyll and Hyde: The Synergistic Effect of Using Divided Source Training Data for Feature Augmentation. In Proceedings of the Sixth International Joint
Conference on Natural Language Processing, pages 623-631, Nagoya, Japan.
Maosong Sun, Dayang Shen, and Benjamin K. Tsou. 1998. Chinese Word Segmentation without Using Lexicon and Hand-crafted Training Data. In 36th Annual Meeting of the Association for Computational Linguistics and 17th International Conference on Computational Linguistics, Volume 2, pages 1265-1271, Montreal, Quebec, Canada.
Weiwei Sun and Jia Xu. 2011. Enhancing Chinese Word Segmentation Using Unlabeled Data. In Proceedings of the Conference on Empirical Methods in Natural Language Processing, pages 970-979.
Yuanhe Tian, Yan Song, Xiang Ao, Fei Xia, Xiaojun Quan, Tong Zhang, and Yonggang Wang. 2020. Joint Chinese Word Segmentation and Part-of-speech Tagging via Two-way Attentions of Autoanalyzed Knowledge. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 8286-8296, Online.
Huihsin Tseng, Pichuan Chang, Galen Andrew, Daniel Jurafsky, and Christopher Manning. 2005. A Conditional Random Field Word Segmenter for Sighan Bakeoff 2005. In Proceedings of the Fourth SIGHAN Workshop on Chinese Language Processing, pages 168-171.
Chunqi Wang and Bo Xu. 2017. Convolutional Neural Network with Word Embeddings for Chinese Word Segmentation. In Proceedings of the Eighth International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 163-172, Taipei, Taiwan.
Deyi Xiong, Min Zhang, and Haizhou Li. 2011. Enhancing Language Models in Statistical Machine Translation with Backward N-grams and Mutual Information Triggers. In Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies, pages 1288-1297, Portland, Oregon, USA.
Jingjing Xu and Xu Sun. 2016. Dependency-based Gated Recursive Neural Network for Chinese Word Segmentation. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 567-572, Berlin, Germany.
Naiwen Xue, Fei Xia, Fu-Dong Chiou, and Marta Palmer. 2005. The Penn Chinese Treebank: Phrase structure annotation of a large corpus. Natural language engineering, 11(2):207-238.
Nianwen Xue and Libin Shen. 2003. Chinese Word Segmentation as LMR Tagging. In Proceedings of the second SIGHAN workshop on Chinese language processing-Volume 17, pages 176-179.