SlowGuess/ACL_2020_OCR huggingface dataset:
Meizhi Ju, Makoto Miwa, and Sophia Ananiadou. 2018. A neural layered model for nested named entity recognition. In Proceedings of NAACL-HLT, pages 1446-1459.
Arzoo Katiyar and Claire Cardie. 2018. Nested named entity recognition revisited. In Proceedings of NAACL-HLT, pages 861-871.
J-D Kim, Tomoko Ohta, Yuka Tateisi, and Junichi Tsujii. 2003. Genia corpus semantically annotated corpus for bio-textmining. Bioinformatics, 19(suppl_1):i180-i182.
D.P. Kingma and J. Ba. 2014. Adam: A method for stochastic optimization. arXiv preprint arXiv: 1412.6980.
Nikita Kitaev, Steven Cao, and Dan Klein. 2019. Multilingual constituency parsing with self-attention and pre-training. In Proceedings of ACL, pages 3499–3505.
Pang Wei Koh and Percy Liang. 2017. Understanding black-box predictions via influence functions. In Proceedings of ICML, pages 1885-1894.
Guillaume Lample, Miguel Ballesteros, Sandeep Subramanian, Kazuya Kawakami, and Chris Dyer. 2016. Neural architectures for named entity recognition. In Proceedings of NAACL-HLT, pages 260-270.
Kenton Lee, Luheng He, Mike Lewis, and Luke Zettle-moyer. 2017. End-to-end neural coreference resolution. In Proceedings of EMNLP, pages 188-197.
Tao Lei, Regina Barzilay, and Tommi Jaakkola. 2016. Rationalizing neural predictions. In Proceedings of EMNLP, pages 107-117.
Tianyu Liu, Jin-Ge Yao, and Chin-Yew Lin. 2019. Towards improving neural named entity recognition with gazetteers. In Proceedings of ACL, pages 5301-5307.
Wei Lu and Dan Roth. 2015. Joint mention extraction and classification with mention hypergraphs. In Proceedings of EMNLP, pages 857-867.
Scott M Lundberg and Su-In Lee. 2017. A unified approach to interpreting model predictions. In Proceedings of NIPS, pages 4765-4774.
Xuezhe Ma and Eduard Hovy. 2016. End-to-end sequence labeling via bi-directional LSTM-CNNs-CRF. In Proceedings of ACL, pages 1064-1074.
Laurens van der Maaten and Geoffrey Hinton. 2008. Visualizing data using t-sne. Journal of machine learning research, 9(Nov):2579-2605.
Aldrian Obaja Muis and Wei Lu. 2017. Labeling gaps between words: Recognizing overlapping mentions with mention separators. In Proceedings of EMNLP, pages 2608-2618.
6457
Makoto Nagao. 1984. A framework of a mechanical translation between Japanese and English by analogy principle. Elsevier Science Publishers.
Joakim Nivre, Johan Hall, and Jens Nilsson. 2004. Memory-based dependency parsing. In Proceedings of CoNLL, pages 49-56, Boston, Massachusetts, USA.
Hiroki Ouchi, Hiroyuki Shindo, and Yuji Matsumoto. 2018. A span selection model for semantic role labeling. In Proceedings of EMNLP, pages 1630-1642.
Razvan Pascanu, Tomas Mikolov, and Yoshua Bengio. 2013. On the difficulty of training recurrent neural networks. In Proceedings of ICML, pages 1310-1318.
Jeffrey Pennington, Richard Socher, and Christopher Manning. 2014. Glove: Global vectors for word representation. In Proceedings of EMNLP), pages 1532-1543.
Gregory Plumb, Denali Molitor, and Ameet S Talwalkar. 2018. Model agnostic supervised local explanations. In Proceedings of NIPS, pages 2515-2524.
Marco Tulio Ribeiro, Sameer Singh, and Carlos Guestrin. 2016. Why should i trust you?: Explaining the predictions of any classifier. In Proceedings of KDD, pages 1135-1144.
Andrew M Saxe, James L McClelland, and Surya Ganguli. 2013. Exact solutions to the nonlinear dynamics of learning in deep linear neural networks. arXiv preprint arXiv:1312.6120.
Mohammad Golam Sohrab and Makoto Miwa. 2018. Deep exhaustive model for nested named entity recognition. In Proceedings of EMNLP, pages 2843-2849.
Nitish Srivastava, Geoffrey Hinton, Alex Krizhevsky, Ilya Sutskever, and Ruslan Salakhutdinov. 2014. Dropout: A simple way to prevent neural networks from overfitting. The Journal of Machine Learning Research, 15(1):1929-1958.
Mitchell Stern, Jacob Andreas, and Dan Klein. 2017. A minimal span-based neural constituency parser. In Proceedings of ACL, pages 818-827.
Erik F. Tjong Kim Sang. 2002. Memory-based named entity recognition. In Proceedings of CoNLL).
Erik F. Tjong Kim Sang and Sabine Buchholz. 2000. Introduction to the CoNLL-2000 shared task chunking. In Proceedings of CoNLL.
Erik F. Tjong Kim Sang and Fien De Meulder. 2003. Introduction to the CoNLL-2003 shared task: Language-independent named entity recognition. In Proceedings of CoNLL, pages 142-147.
Bailin Wang and Wei Lu. 2018. Neural segmental hypergraphs for overlapping mention recognition. In Proceedings of EMNLP, pages 204-214.
Wenhui Wang and Baobao Chang. 2016. Graph-based dependency parsing with bidirectional LSTM. In Proceedings of ACL, pages 2306-2315.
Sam Wiseman and Karl Stratos. 2019. Label-agnostic sequence labeling by copying nearest neighbors. In Proceedings of ACL, pages 5363-5369.
Congying Xia, Chenwei Zhang, Tao Yang, Yaliang Li, Nan Du, Xian Wu, Wei Fan, Fenglong Ma, and Philip Yu. 2019. Multi-grained named entity recognition. In Proceedings of ACL, pages 1430-1440.
Changmeng Zheng, Yi Cai, Jingyun Xu, Ho-fung Leung, and Guandong Xu. 2019. A boundary-aware neural model for nested named entity recognition. In Proceedings of EMNLP-IJCNLP, pages 357-366.
6458
Name	Value
CNN window size	3
CNN filters	30
BiLSTM layers	2
BiLSTM hidden units	100 dimensions
Mini-batch size	8
Optimization	Adam
Learning rate	0.001
Dropout ratio	{0.1, 0.3, 0.5}
Table 5: Hyperparameters used in the experiments.
Network setup Basically, we follow the encoder architecture proposed by Ma and Hovy (2016). First, the token-encoding layer encodes each token of the input sentence  $w_{t} \in (w_{1}, w_{2}, \ldots, w_{T})$  to a sequence of the vector representations  $\mathbf{w}_{1:T} = (\mathbf{w}_{1}, \mathbf{w}_{2}, \ldots, \mathbf{w}_{T})$ . For the models using GloVe, we use the GloVe 100-dimensional embeddings<sup>13</sup> (Pennington et al., 2014) and character-level CNN. For the models using BERT, we use the BERT-Base, Cased<sup>14</sup> (Devlin et al., 2019), where we use the first subword embeddings within each token in the last layer of BERT. During training, we fix the word embeddings (except the CNN). Then, the encoded token representations  $\mathbf{w}_{1:T} = (\mathbf{w}_{1}, \mathbf{w}_{2}, \ldots, \mathbf{w}_{T})$  are fed to bidirectional LSTM (BiLSTM) (Graves et al., 2013) for computing contextual ones  $\vec{\mathbf{h}}_{1:T}$  and  $\vec{\mathbf{h}}_{1:T}$ . We use 2 layers of the stacked BiLSTMs (2 forward and 2 backward LSTMs) with 100-dimensional hidden units. From  $\vec{\mathbf{h}}_{1:T}$  and  $\vec{\mathbf{h}}_{1:T}$ , we create  $\mathbf{h}_s^{\mathrm{lstm}}$  for each span  $s = (a,b)$  based on LSTM-minus (Wang and Chang, 2016). For flat NER, we use the representation  $\mathbf{h}_s^{\mathrm{lstm}} = [\vec{\mathbf{h}}_b - \vec{\mathbf{h}}_{a-1}, \vec{\mathbf{h}}_a - \vec{\mathbf{h}}_{b+1}]$ . For nested NER, we use  $\mathbf{h}_s^{\mathrm{lstm}} = [\vec{\mathbf{h}}_b - \vec{\mathbf{h}}_{a-1}, \vec{\mathbf{h}}_a - \vec{\mathbf{h}}_{b+1}, \vec{\mathbf{h}}_a + \vec{\mathbf{h}}_b, \vec{\mathbf{h}}_a + \vec{\mathbf{h}}_b]$ . We then multiply  $\mathbf{h}_s^{\mathrm{lstm}}$  with a weight matrix  $\mathbf{W}$  and obtain the span representation:  $\mathbf{h}_s = \mathbf{W} \mathbf{h}_s^{\mathrm{lstm}}$ . Finally, we use the span representation  $\mathbf{h}_s$  for computing the label distribution in each model. For efficient computation, following Sohrab and Miwa (2018), we enumerate all possible spans in a sentence with the sizes less than or equal to the maximum span size  $L$ , i.e., each span  $s = (a,b)$  is satisfied with the condition  $b - a < L$ . We set  $L$  as 6.
$^{13}$ https://nlp.stanford.edu/projects/glove/ $^{14}$ https://github.com/google-research/bert
Hyperparameters Table 5 lists the hyperparameters used in the experiments. We initialize all the parameter matrices in BiLSTMs with random orthonormal matrices (Saxe et al., 2013). Other parameters are initialized following Glorot and Bengio (2010). We apply dropout (Srivastava et al., 2014) to the token-encoding layer and the input vectors of each LSTM with dropout ratio of  $\{0.1, 0.3, 0.5\}$ .
Optimization To optimize the parameters, we use Adam (Kingma and Ba, 2014) with  $\beta_{1} = 0.9$  and  $\beta_{2} = 0.999$ . The initial learning rate is set to  $\eta_0 = 0.001$ . The learning rate is updated on each epoch as  $\eta_t = \eta_0 / (1 + \rho t)$ , where the decay rate is  $\rho = 0.05$  and  $t$  is the number of epoch completed. A gradient clipping value is set to 5.0 (Pascanu et al., 2013). Parameter updates are performed in mini-batches of 8. The number of training epochs is set to 100. We save the parameters that achieve the best F1 score on each development set and evaluated them on each test set. Training the models takes less than one day on a single GPU, NVIDIA DGX-1 with Tesla V100.
A.2 Feature space visualization
(a) Classifier-based
(b) Instance-based
Figure 3: Visualization of entity span features computed by classifier-based and instance-based models.
To better understand span representations learned by our method, we observe the feature space. Specifically, we visualize the span representations  $\mathbf{h}_s$  on the CoNLL-2003 development set. Figure 3 visualizes two-dimensional entity span representations by t-distributed Stochastic Neighbor Embedding (t-SNE) (Maaten and Hinton, 2008). Both models successfully learned feature spaces where the instances with the same label come close each other.
6459