SlowGuess/NAACL_2021_OCR huggingface dataset:
Rui Wang, Xin Xin, Wei Chang, Kun Ming, Biao Li, and Xin Fan. 2019. Chinese ner with height-limited constituent parsing. In Proceedings of AAAI.
Ralph Weischedel, Martha Palmer, Mitchell Marcus, Eduard Hovy, Sameer Pradhan, Lance Ramshaw, Ni-anwen Xue, Ann Taylor, Jeff Kaufman, Michelle Franchini, et al. 2013. Ontonotes release 5.0 ldc2013t19. Linguistic Data Consortium.
Tsung-Hsien Wen, Milica Gasic, Nikola Mrksic, Pei hao Su, David Vandyke, and Steve J. Young. 2015. Semantically conditioned lstm-based natural language generation for spoken dialogue systems. In Proceedings of EMNLP.
Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, R'emi Louf, Morgan Funtowicz, and Jamie Brew. 2019. Huggingface's transformers: State-of-the-art natural language processing. ArXiv.
Han Xiao. 2018. bert-as-service. https://github.com/hanxiao/bert-as-service.
Juntao Yu, Bernd Bohnet, and Massimo Poesio. 2020. Named entity recognition as dependency parsing. In Proceedings of ACL.
Yue Zhang, Qi Liu, and Linfeng Song. 2018a. Sentence-state lstm for text representation. In Proceedings of ACL.
Yue Zhang and Jie Yang. 2018. Chinese NER using lattice LSTM. In Proceedings of ACL.
Yuhao Zhang, Peng Qi, and Christopher D. Manning. 2018b. Graph convolution over pruned dependency trees improves relation extraction. In Proceedings of EMNLP.
We test our model on RTX 2080 Ti GPU and Nvidia Tesla V100 GPU, with CUDA version 10.1, PyTorch version 1.40. The average run time for Syn-LSTM is 52 sec/epoch, 55 sec/epoch, 290 sec/epoch, 350 sec/epoch for Catalan, Spanish, Chinese and English datasets respectively. The total number of parameters is 11M. Table 10 shows the performance of our model on the dev sets of OntoNotes 5.0 English and Chinese, SemEval 2010 Task 1 Catalan and Spanish.
For hyper-parameter, we use the FastText (Grave et al., 2018) 300 dimensional embeddings to initialize the word embeddings for Catalan, Spanish, and Chinese. For OntoNotes 5.0 English, we adopt the publicly available GloVE (Pennington et al., 2014) 100 dimensional embeddings to initialize the word embeddings. For experiments with the contextualized representation, we adopt the pre-trained
English	Chinese	Catalan	Spanish
Dependency LAS†	94.89	89.28	93.25	93.35
Table 8: Performance of the trained dependency parser.
Dataset	Entity Length
1	2	3	4	5	≥6
English	Train	46,525	17,391	9,714	4,892	1,938	1,368
Dev	6,325	2,395	1,256	643	275	172
Test	6,129	2,598	1,359	706	278	187
Chinese	Train	47,285	9,668	3,626	1,139	467	358
Dev	6,969	1,397	473	169	55	41
Test	5,479	1299	473	146	55	42
Catalan	Train	8,819	3,897	1,742	264	119	437
Dev	1,370	676	269	40	18	58
Test	1,601	811	338	57	27	76
Spanish	Train	10,307	3,609	2,302	301	175	603
Dev	1,523	559	348	54	31	100
Test	1,755	702	369	59	34	127
Table 9: Number of entities with respect to entity length for OntoNotes 5.0 English and Chinese, SemEval 2010 Catalan and Spanish datasets.
language model BERT (Devlin et al., 2019) for the four datasets. Specifically, we use bert-as-service (Xiao, 2018) to generate the contextualized word representation without fine-tuning. Following Luo et al. (2020), we select the  $18^{th}$  layer of the cased version of BERT large model for the experiments on the OntoNotes 5.0 English data. We use the  $9^{th}$  layer of cased version of BERT base model for the experiments on the rest three datasets. For the character embedding, we randomly initialize the character embeddings and set the dimension as 30, and set the hidden size of character-level BiLSTM as 50. The hidden size of GCN and SynLSTM is set as 200. Note that we only use one layer of bi-directional Syn-LSTM for our experiments. Dropout is set to 0.5 for input embeddings and hidden states. We adopt stochastic gradient descent (SGD) to optimize our model with batch size 100, L2 regularization  $10^{-8}$ , learning rate 0.2 and the learning rate is decayed with respect to the number of epoch<sup>9</sup>.
B Performance of dependency parser
Table 8 presents the performance of dependency parser.
Figure 7: Left: English, Right: Chinese. The x-axis indicates the value of gate  $m_{t}$ , the y-axis denotes the number of cells.
Figure 8: Left: English, Right: Chinese.  $x$ -axis: sentence length.  $y$ -axis:  $F_{1}$  score (\%). Note that DGLSTM-CRF $_{+\mathrm{ELMO}}$  have better performance compared to DGLSTM-CRF $_{+\mathrm{BERT}}$  based on the results in the main paper.
C More data statistics
Table 9 shows the statistics of the number of entities with respect to entity length for OntoNotes 5.0 English and Chinese, SemEval 2010 Task 1 Catalan and Spanish datasets.
D More Robustness Analysis
Figure 7 shows the comparisons of the models of using random trees and given trees on OntoNotes 5.0 English and Chinese datasets.
E Effect of Sentence Length
We compare the performance of our Syn-LSTM-CRF $_{+\text{BERT}}$  with BiLSTM-CRF $_{+\text{BERT}}$  and DGLSTM-CRF $_{+\text{ELMO}}$  models with respect to sentence length, and the results are shown in Figure 8.
We further show an example to visualize the propagation of non-local information (Figure 9). The example is selected from OntoNotes 5.0 English dataset. Even though the DGLSTM-CRF (Jie and Lu, 2019) model is able to recognize "Tianshui" as a named entity, it predicts a wrong entity type as PERSON while the true type is GPE. If only looking at the first half of the sentence, it is possible to predict "Tianshui" as PERSON because of the local information "age". However, the second half of the sentence confirms that the entity type of
<sup>9</sup>We set the decay as 0.1 and the learning rate for each epoch equals to learning_rate/(1 + decay * (epoch - 1)).
Models	English	Chinese	Catalan	Spanish
P.	R.	F1	P.	R.	F1	P.	R.	F1	P.	.R.	F1
Syn-LSTM-CRF	86.73	87.71	87.22	77.25	75.74	76.49	84.48	82.60	83.53	83.76	82.22	82.98
Syn-LSTM-CRF+BERT	88.10	90.27	89.17	78.05	78.84	78.45	89.87	89.76	89.81	88.50	88.60	88.55
Table 10: Experimental results [%] on dev set.
Figure 9: An example of dependency tree. The mentioned entity is highlighted in orange, and the entity type is GPE.
"Tianshui" is GPE. With the non-local information from the graph-encoded representation, our SynLSTM-CRF successfully predicts the right entity type.