SlowGuess/ACL_2020_OCR huggingface dataset:
To further illustrate the efficiency of the SoftLexicon method, we also conducted an experiment to evaluate its inference speed against sentences of different lengths, as shown in Table 4. For a fair comparison, we set the batch size to 1 in all of the compared methods. The results show that the proposed method achieves significant improvement in speed over Lattice-LSTM and LR-CNN when processing short sentences. With the increase of sentence length, the proposed method is consistently faster than Lattice-LSTM and LR-CNN despite the speed degradation due to the recurrent architecture of LSTM. Overall, the proposed SoftLexicon method shows a great advantage over other methods in computational efficiency.
4.3 Effectiveness Study
Tables  $3 - 6^{3}$  show the performances of our method against the compared baselines. In this study, the sequence modeling layer of our method was
2Please refer to the attached source code for more implementation detail of this work and access https:// github.com/jiesutd/LatticeLSTM for pre-trained word and character embeddings.
3In Table 3-5, * indicates that the model uses external labeled data for semi-supervised learning. † means that the model also uses discrete features.
5956
Input	Models	P	R	F1
Gold seg	Yang et al., 2016	65.59	71.84	68.57
Yang et al., 2016*†	72.98	80.15	76.40
Che et al., 2013*	77.71	72.51	75.02
Wang et al., 2013*	76.43	72.32	74.32
Word-based (LSTM)	76.66	63.60	69.52
+ char + bichar	78.62	73.13	75.77
Auto seg	Word-based (LSTM)	72.84	59.72	65.63
+ char + bichar	73.36	70.12	71.70
No seg	Char-based (LSTM)	68.79	60.35	64.30
+ bichar + softword	74.36	69.43	71.89
+ ExSoftword	69.90	66.46	68.13
+ bichar + ExSoftword	73.80	71.05	72.40
Lattice-LSTM	76.35	71.56	73.88
LR-CNN (Gui et al., 2019)	76.40	72.60	74.45
SoftLexicon (LSTM)	77.28	74.07	75.64
SoftLexicon (LSTM) + bichar	77.13	75.22	76.16
BERT-Tagger	76.01	79.96	77.93
BERT + LSTM + CRF	81.99	81.65	81.82
SoftLexicon (LSTM) + BERT	83.41	82.21	82.81
implemented with a single layer bidirectional LSTM.
OntoNotes. Table 3 shows results  ${}^{4}$  on the OntoNotes dataset, where gold word segmentation is provided for both training and testing data. The methods of the "Gold seg" and the "Auto seg" groups are all word-based, with the former input building on gold word segmentation results and the latter building on automatic word segmentation results by a segmenter trained on OntoNotes training data. The methods used in the "No seg" group are character-based. From the table, we can make several observations. First, when gold word segmentation was replaced by automatically generated word segmentation, the F1 score decreases from  ${75.77}\%$  to  ${71.70}\%$  . This reveals the problem of treating the predicted word segmentation result as the true result in the word-based Chinese NER. Second, the F1 score of the Char-based (LSTM)+ExSoftword model is greatly improved from that of the Char-based (LSTM) model. This indicates the feasibility of the naive ExSoftword method. However, it still greatly underperforms relative to Lattice-LSTM, which reveals its deficiency in utilizing word information. Lastly, the proposed SoftLexicon method outperforms Lattice-LSTM by 1.76% with respect to the F1 score, and obtains a greater improvement of 2.28% combining the bichar
Table 3: Performance on OntoNotes. A model followed by (LSTM) (e.g., Proposed (LSTM)) indicates that its sequence modeling layer is LSTM-based.
Models	P	R	F1
Chen et al., 2006	91.22	81.71	86.20
Zhang et al. 2006*	92.20	90.18	91.18
Zhou et al. 2013	91.86	88.75	90.28
Lu et al. 2016	-	-	87.94
Dong et al. 2016	91.28	90.62	90.95
Char-based (LSTM)	90.74	86.96	88.81
+ bichar+softword	92.97	90.80	91.87
+ ExSoftword	90.77	87.23	88.97
+ bichar+ExSoftword	93.21	91.57	92.38
Lattice-LSTM	93.57	92.79	93.18
LR-CNN (Gui et al., 2019)	94.50	92.93	93.71
SoftLexicon (LSTM)	94.63	92.70	93.66
SoftLexicon (LSTM) + bichar	94.73	93.40	94.06
BERT-Tagger	93.40	94.12	93.76
BERT + LSTM + CRF	95.06	94.61	94.83
SoftLexicon (LSTM) + BERT	95.75	95.10	95.42
Table 4: Performance on MSRA.
Models	NE	NM	Overall
Peng and Dredze, 2015	51.96	61.05	56.05
Peng and Dredze, 2016*	55.28	62.97	58.99
He and Sun, 2017a	50.60	59.32	54.82
He and Sun, 2017b*	54.50	62.17	58.23
Char-based (LSTM)	46.11	55.29	52.77
+ bichar+softword	50.55	60.11	56.75
+ ExSoftword	44.65	55.19	52.42
+ bichar+ExSoftword	58.93	53.38	56.02
Lattice-LSTM	53.04	62.25	58.79
LR-CNN (Gui et al., 2019)	57.14	66.67	59.92
SoftLexicon (LSTM)	59.08	62.22	61.42
SoftLexicon (LSTM) + bichar	58.12	64.20	59.81
BERT-Tagger	65.77	62.05	63.80
BERT + LSTM + CRF	69.65	64.62	67.33
SoftLexicon (LSTM) + BERT	70.94	67.02	70.50
Table 5: Performance on Weibo. NE, NM and Overall denote F1 scores for named entities, nominal entities (excluding named entities) and both, respectively.
feature. It even performs comparably with the word-based methods of the "Gold seg" group, verifying its effectiveness on OntoNotes.
MSRA/Weibo/Resume. Tables 4, 5 and 6 show results on the MSRA, Weibo and Resume datasets, respectively. Compared methods include the best statistical models on these data set, which leveraged rich handcrafted features (Chen et al., 2006; Zhang et al., 2006; Zhou et al., 2013), character embedding features (Lu et al., 2016; Peng and Dredze, 2016), radical features (Dong et al., 2016), cross-domain data, and semi-supervised data (He and Sun, 2017b). From the tables, we can see that the performance of the proposed Soft-lexion method is significant better than that of Lattice-LSTM and other baseline methods on all three datasets.
4 A result in boldface indicates that it is statistically significantly better  $(p < 0.01$  in pairwise  $t$ -test) than the others in the same box.
5957
Models	P	R	F1
Word-based (LSTM)	93.72	93.44	93.58
+char+bichar	94.07	94.42	94.24
Char-based (LSTM)	93.66	93.31	93.48
+ bichar+softword	94.53	94.29	94.41
+ ExSoftword	95.29	94.42	94.85
+ bichar+ExSoftword	96.14	94.72	95.43
Lattice-LSTM	94.81	94.11	94.46
LR-CNN (Gui et al., 2019)	95.37	94.84	95.11
SoftLexicon (LSTM)	95.30	95.77	95.53
SoftLexicon (LSTM) + bichar	95.71	95.77	95.74
BERT-Tagger	94.87	96.50	95.68
BERT + LSTM + CRF	95.75	95.28	95.51
SoftLexicon (LSTM) + BERT	96.08	96.13	96.11
Table 6: Performance on Resume.
Models	OntoNotes	MSRA	Weibo	Resume
SoftLexicon (LSTM)	75.64	93.66	61.42	95.53
ExSoftword (CNN)	68.11	90.02	53.93	94.49
SoftLexicon (CNN)	74.08	92.19	59.65	95.02
ExSoftword (Transformer)	64.29	86.29	52.86	93.78
SoftLexicon (Transformer)	71.21	90.48	61.04	94.59
Table 7: F1 score with different implementations of the sequence modeling layer. ExSoftword is the shorthand of Char-based+bichar+ExSoftword.
4.4 Transferability Study
Table 7 shows the performance of the SoftLexicon method when implementing the sequence modeling layer with different neural architecture. From the table, we can first see that the LSTM-based architecture performed better than the CNN- and transformer- based architectures. In addition, our method with different sequence modeling layers consistently outperformed their corresponding Ex-Softword baselines. This confirms the superiority of our method in modeling lexicon information in different neural NER models.
4.5 Combining Pre-trained Model
We also conducted experiments on the four datasets to further verify the effectiveness of SoftLexicon in combination with pre-trained model, the results of which are shown in Tables 3-6. In these experiments, we first use a BERT encoder to obtain the contextual representations of each sequenc, and then concatenated them into the character representations. From the table, we can see that the SoftLexicon method with BERT outperforms the BERT tagger on all four datasets. These results show that the SoftLexicon method can be effectively combined with pre-trained model. Moreover, the results also verify the effectiveness of our method in utilizing lexicon information,
Models	OntoNotes	MSRA	Weibo	Resume
SoftLexicon (LSTM)	75.64	93.66	61.42	95.53
- “M” group	75.06	93.09	58.13	94.72
- Distinction	70.29	92.08	54.85	94.30
- Weighted pooling	72.57	92.76	57.72	95.33
- Overall weighting	74.28	93.16	59.55	94.92
Table 8: An ablation study of the proposed model.
which means it can complement the information obtained from the pre-trained model.
To investigate the contribution of each component of our method, we conducted ablation experiments on all four datasets, as shown in table 8.
(1) In Lattice-LSTM, each character receives word information only from the words that begin or end with it. Thus, the information of the words that contain the character inside is ignored. However, the SoftLexicon prevents the loss of this information by incorporating the "Middle" group of words. In the " - 'M' group" experiment, we removed the "Middle" group in SoftLexicon, as in Lattice-LSTM. The degradation in performance on all four datasets indicates the importance of the "M" group of words, and confirms the advantage of our method.
(2) Our method proposed to draw a clear distinction between the four "BMES" categories of matched words. To study the relative contribution of this design, we conducted experiments to remove this distinction, i.e., we simply added up all the weighted words regardless of their categories. The decline in performance verifies the significance of a clear distinction for different matched words.
(3) We proposed two strategies for pooling the four word sets in Section 3.2. In the “- Weighted pooling” experiment, the weighted pooling strategy was replaced with mean-pooling, which degrades the performance. Compared with mean-pooling, the weighting strategy not only succeeds in weighing different words by their significance, but also introduces the frequency information of each word in the statistical data, which is verified to be helpful.
(4) Although existing lexicon-based methods like Lattice-LSTM also use word weighting, unlike the proposed Soft-lexion method, they fail to perform weight normalization among all the matched words. For example, Lattice-LSTM only normalizes the weights inside the "B" group or the "E" group. In the "- Overall weighting" experiment, we performed weight normalization inside each
5958
"BMES" group as Lattice-LSTM does, and found the resulting performance to be degraded. This result shows that the ability to perform overall weight normalization among all matched words is also an advantage of our method.
In this work, we addressed the computational efficiency of utilizing word lexicons in Chinese NER. To obtain a high-performing Chinese NER system with a fast inference speed, we proposed a novel method to incorporate the lexicon information into the character representations. Experimental studies on four benchmark Chinese NER datasets reveal that our method can achieve a much faster inference speed and better performance than the compared state-of-the-art methods.
The authors wish to thank the anonymous reviewers for their helpful comments. This work was partially funded by China National Key RD Program (No. 2018YFB1005104, 2018YFC0831105, 2017YFB1002104), National Natural Science Foundation of China (No. 61976056, 61532011, 61751201), Shanghai Municipal Science and Technology Major Project (No.2018SHZDZX01), Science and Technology Commission of Shanghai Municipality Grant (No.18DZ1201000, 16JC1420401, 17JC1420200).
Wanxiang Che, Mengqiu Wang, Christopher D Manning, and Ting Liu. 2013. Named entity recognition with bilingual constraints. In *NAACL*, pages 52-62.
Aitao Chen, Fuchun Peng, Roy Shan, and Gordon Sun. 2006. Chinese named entity recognition with conditional probabilistic models. In SIGHAN Workshop on Chinese Language Processing.
Yubo Chen, Liheng Xu, Kang Liu, Daojian Zeng, and Jun Zhao. 2015. Event extraction via dynamic multi-pooling convolutional neural networks. In ACL—IJCNLP, volume 1, pages 167-176.
Jason Chiu and Eric Nichols. 2016. Named entity recognition with bidirectional LSTM-cnns. Transactions of the Association of Computational Linguistics, 4(1):357-370.
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2018. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805.
Dennis Diefenbach, Vanessa Lopez, Kamal Singh, and Pierre Maret. 2018. Core techniques of question answering systems over knowledge bases: a survey. KAIS, 55(3):529-569.
Ruixue Ding, Pengjun Xie, Xiaoyan Zhang, Wei Lu, Linlin Li, and Luo Si. 2019. A neural multidigraph model for chinese ner with gazetteers. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 1462-1467.
Chuanhai Dong, Jiajun Zhang, Chengqing Zong, Masanori Hattori, and Hui Di. 2016. Character-based LSTM-crf with radical-level features for chinese named entity recognition. In *Natural Language Understanding and Intelligent Applications*, pages 239–250. Springer.
G David Forney. 1973. The viterbi algorithm. Proceedings of the IEEE, 61(3):268-278.
Tao Gui, Ruotian Ma, Qi Zhang, Lujun Zhao, Yu-Gang Jiang, and Xuanjing Huang. 2019a. Cnn-based chinese ner with lexicon rethinking. In Proceedings of the 28th International Joint Conference on Artificial Intelligence, pages 4982-4988. AAAI Press.
Tao Gui, Yicheng Zou, Qi Zhang, Minlong Peng, Jinlan Fu, Zhongyu Wei, and Xuan-Jing Huang. 2019b. A lexicon-based graph neural network for chinese ner. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 1039-1049.
Hangfeng He and Xu Sun. 2017a. F-score driven max margin neural network for named entity recognition in chinese social media. In Proceedings of the 15th Conference of the European Chapter of the Association for Computational Linguistics: Volume 2, Short Papers, pages 713-718.
Hangfeng He and Xu Sun. 2017b. A unified model for cross-domain and semi-supervised named entity recognition in chinese social media. In AAAI.
Jingzhou He and Houfeng Wang. 2008. Chinese named entity recognition and word segmentation based on character. In Proceedings of the Sixth SIGHAN Workshop on Chinese Language Processing.
Zhiheng Huang, Wei Xu, and Kai Yu. 2015. Bidirectional lstm- crf models for sequence tagging. arXiv preprint arXiv:1508.01991.
Diederik P Kingma and Jimmy Ba. 2014. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980.
John Lafferty, Andrew McCallum, and Fernando CN Pereira. 2001. Conditional random fields: Probabilistic models for segmenting and labeling sequence data.
5959
Gina-Anne Levow. 2006. The third international chinese language processing bakeoff: Word segmentation and named entity recognition. In SIGHAN Workshop on Chinese Language Processing, pages 108-117.
Haibo Li, Masato Hagiwara, Qi Li, and Heng Ji. 2014. Comparison of the impact of word segmentation on name tagging for chinese and japanese. In LREC, pages 2532-2536.
Liyuan Liu, Jingbo Shang, Xiang Ren, Frank Xu, Huan Gui, Jian Peng, and Jiawei Han. 2018. Empower sequence labeling with task-aware neural language model. AAAI Conference on Artificial Intelligence.
Wei Liu, Tongge Xu, Qinghua Xu, Jiayu Song, and Yueran Zu. 2019. An encoding strategy based word-character LSTM for Chinese ner. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 2379-2389.
Zhangxun Liu, Conghui Zhu, and Tiejun Zhao. 2010. Chinese named entity recognition with a sequence labeling approach: based on characters, or based on words? In Advanced intelligent computing theories and applications. With aspects of artificial intelligence, pages 634-640. Springer.
Yanan Lu, Yue Zhang, and Dong-Hong Ji. 2016. Multi-prototype chinese character embedding. In LREC.
Nanyun Peng and Mark Dredze. 2015. Named entity recognition for chinese social media with jointly trained embeddings. In EMNLP.
Nanyun Peng and Mark Dredze. 2016. Improving named entity recognition for chinese social media with word segmentation representation learning. In ACL, page 149.
Sebastian Riedel, Limin Yao, Andrew McCallum, and Benjamin M Marlin. 2013. Relation extraction with matrix factorization and universal schemas. In Proceedings of the 2013 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 74-84.
Dianbo Sui, Yubo Chen, Kang Liu, Jun Zhao, and Shengping Liu. 2019. Leverage lexical knowledge for chinese named entity recognition via collaborative graph network. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 3821-3831.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In Advances in Neural Information Processing Systems, pages 5998-6008.
Mengqiu Wang, Wanxiang Che, and Christopher D. Manning. 2013. Effective bilingual constraints for semi-supervised learning of named entity recognizers. In AAAI.
Ralph Weischedel, Sameer Pradhan, Lance Ramshaw, Martha Palmer, Nianwen Xue, Mitchell Marcus, Ann Taylor, Craig Greenberg, Eduard Hovy, Robert Belvin, et al. 2011. Ontonotes release 4.0. LDC2011T03, Philadelphia, Penn.: Linguistic Data Consortium.
Jie Yang, Zhiyang Teng, Meishan Zhang, and Yue Zhang. 2016. Combining discrete and neural features for sequence labeling. In CICling. Springer.
Suxiang Zhang, Ying Qin, Juan Wen, and Xiaojie Wang. 2006. Word segmentation and named entity recognition for sighan bakeoff3. In SIGHAN Workshop on Chinese Language Processing, pages 158-161.
Yue Zhang and Jie Yang. 2018. Chinese ner using lattice lstm. Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (ACL), 1554-1564.
Hai Zhao and Chunyu Kit. 2008. Unsupervised segmentation helps supervised learning of character tagging for word segmentation and named entity recognition. In Proceedings of the Sixth SIGHAN Workshop on Chinese Language Processing.
Junsheng Zhou, Weiguang Qu, and Fen Zhang. 2013. Chinese named entity recognition via joint identification and categorization. Chinese journal of electronics, 22(2):225-230.