SlowGuess/ACL_2020_OCR huggingface dataset:
The AIDA entity dataset has a 3-level ontology, termed type, subtype, and subsubtype. A mention can only have one label for each level, hence the dataset is single-path, thus the branching factors  $(k_{1}, k_{2}, k_{3})$  for the three layers are set to  $(1, 1, 1)$ .
BBN Weischedel and Brunstein (2005) labeled a portion of the one million word Penn Treebank corpus of Wall Street Journal texts (LDC95T7) using a two-level hierarchy, resulting in the BBN Pronoun Coreference and Entity Type Corpus. We follow the train/test split by Ren et al. (2016b), and follow the train/dev split by Zhang et al. (2018).
OntoNotes Gillick et al. (2014) sampled sentences from the OntoNotes corpus and annotated the entities using 89 types. We follow the train/dev/test data split by Shimaoka et al. (2017).
7 For the OntoNotes dataset, this also includes the per-level threshold  $\delta_{\mathrm{lev}(k)}$
8470
Dataset	Train	Dev	Test	#Levels	#Types	Multi-path?	α	β	λ	pD	k1,...,L
AIDA	2,492	558	1,383	3	187	single-path	0.1	0.3	0.1	0.5	(1,1,1)
BBN	84,078	2,000	13,766	2	56	multi-path	0.2	0.1	0.003	0.5	(2,1)
OntoNotes	251,039	2,202	8,963	3	89	multi-path	0.15	0.1	0.001	0.5	(2,1,1)
FIGER	2,000,000	10,000	563	2	113	multi-path	0.2	0.1	0.0001	0.5	(2,1)
FIGER Ling and Weld (2012) sampled a dataset from Wikipedia articles and news reports. Entity mentions in these texts are mapped to a 113-type ontology derived from Freebase (Bollacker et al., 2008). Again, we follow the data split by Shimaoka et al. (2017).
The statistics of these datasets and their accompanying ontologies are listed in Table 1, together with their respective hyperparameters.
To best compare to recent prior work, we follow Lin and Ji (2019) where the ELMo encodings of words are fixed and not updated. We use all 3 layers of ELMo output, so the initial embedding has dimension  $d_w = 3072$ . We set the type embedding dimensionality to be  $d_t = 1024$ . The initial learning rate is  $10^{-5}$  and the batch size is 256.
Hyperparameter choices are tuned on dev sets, and are listed in Table 1. We employ early stopping: choosing the model that yields the best micro  $F_{1}$  score on dev sets.
Our models are implemented using AllenNLP (Gardner et al., 2018), with implementation for subtyping relation constraints from OpenKE (Han et al., 2018).
5.3 Baselines
We compare our approach to major prior work in FET that are capable of multi-path entity typing. For AIDA, since there are no prior work on this dataset to our knowledge, we also implemented multi-label classification as a set of binary classifier models (similar to Lin and Ji (2019)) as a baseline, with our mention feature extractor. The results are shown in Table 2 as "Multi-label".
We follow prior work and use strict accuracy (Acc), macro  $F_{1}$  (MaF), and micro  $F_{1}$  (MiF) scores. Given instance  $x_{i}$ , we denote the gold type set as  $Y_{i}$  and the predicted type set  $\hat{Y}_i$ . The strict accuracy is the ratio of instances where  $Y_{i} = \hat{Y}_{i}$ . Macro  $F_{1}$  is the average of all  $F_{1}$  scores between  $Y_{i}$  and  $\hat{Y}_i$  for all instances, whereas micro  $F_{1}$  counts total true positives, false negatives and false positives globally.
We also investigate per-level accuracies on AIDA. The accuracy on level  $l$  is the ratio of instances whose predicted type set and gold type set are identical at level  $l$ . If there is no type output at level  $l$ , we append with OTHER to create a dummy type at level  $l$ : e.g. /person/OTHER/OTHER. Hence accuracy of the last level (in AIDA, level 3) is equal to the strict accuracy.
5.5 Results and Discussions
All our results are run under the two conditions regarding partial type paths: exclusive or undefined. The result of the AIDA dataset is shown in Table 2. Our model under the exclusive case outperforms a multi-label classification baseline over all metrics.
Of the 187 types specified in the AIDA ontology, the train/dev set only covers 93 types. The test set covers 85 types, of which 63 are seen types. We could perform zero-shot entity typing by initializing a type's embedding using the type name (e.g. /fac/structure/plaza) together with its description (e.g. "An open urban public space, such as a city square") as is designated in the data annotation manual. We leave this as future work.
Table 1: Statistics of various datasets and their corresponding hyperparameter settings.
Approach	L1	L2	L3	MaF	MiF
Ours (exclusive)	81.6	43.1	32.0	60.6	60.0
Ours (undefined)	80.0	43.3	30.2	59.3	58.0
- Subtyping constraints	80.3	40.9	29.9	59.1	58.3
- Multi-level margins	76.9	40.2	29.8	57.4	56.9
Multi-label	80.5	42.1	30.7	59.7	57.9
Table 2: Results on the AIDA dataset.
8 The OntoNotes dataset has an additional set of hyperparameters, i.e. the per-level threshold  $\delta_{1,2,3} = (2.5, 3.0, 0.0)$ .
9 Zhang et al. (2018) included document-level information in their best results—for fair comparison, we used their results without document context, as are reported in their ablation tests.
8471
Approach	BBN	OntoNotes	FIGER
Acc	MaF	MiF	Acc	MaF	MiF	Acc	MaF	MiF
Ling and Weld (2012)	46.7	67.2	61.2	-†	52.3	69.9	69.3
Ren et al. (2016b)	49.4	68.8	64.5	51.6	67.4	62.4	49.4	68.8	64.5
Ren et al. (2016a)	67.0	72.7	73.5	55.1	71.1	64.7	53.3	69.3	66.4
Abhishek et al. (2017)	60.4	74.1	75.7	52.2	68.5	63.3	59.0	78.0	74.9
Shimaoka et al. (2017)	-†	51.7	71.0	64.9	59.7	79.0	75.4
Murty et al. (2018)	-†	-†	59.7	78.3	75.4
Zhang et al. (2018)	58.1	75.7	75.1	53.2	72.1	66.5	60.2‡	78.7‡	75.5‡
Lin and Ji (2019)	55.9	79.3	78.1	63.8*	82.9*	77.3*	62.9	83.0	79.8
Ours (exclusive)	48.2	63.2	61.0	58.3	72.4	67.2	69.1	82.6	80.8
Ours (undefined)	75.2	79.7	80.5	58.7	73.0	68.1	65.5	80.5	78.1
- Subtyping constraint	73.2	77.8	78.4	58.3	72.2	67.1	65.4	81.4	79.2
- Multi-level margins	68.9	73.2	74.2	58.5	71.7	66.0	68.1	80.4	78.0
†: Not run on the specific dataset; *: Not strictly comparable due to non-standard, much larger training set;
$\ddagger$ : Result has document-level context information, hence not comparable.
Table 3: Results of common FET datasets: BBN, OntoNotes, and FIGER. Numbers in italic are results obtained with various augmentation techniques, either larger data or larger context, hence not directly comparable.
Results for the BBN, OntoNotes, and FIGER can be found in Table 3. Across 3 datasets, our method produces the state-of-the-art performance on strict accuracy and micro  $F_{1}$  scores, and state-of-the-art or comparable (±0.5%) performance on macro  $F_{1}$  score, as compared to prior models, e.g. (Lin and Ji, 2019). Especially, our method improves upon the strict accuracy substantially (4%–8%) across these datasets, showing our decoder are better at outputting exact correct type sets.
Partially type paths: exclusive or undefined? Interestingly, we found that for AIDA and FIGER, partial type paths should be better considered as exclusive, whereas for BBN and OntoNotes, considering them as undefined leads to better performance. We hypothesize that this comes from how the data is annotated—the annotation manual may contain directives as whether to interpret partial type paths as exclusive or undefined, or the data may be non-exhaustively annotated, leading to undefined partial types. We advocate for careful investigation into partial type paths for future experiments and data curation.
Ablation Studies We compare our best model with various components of our model removed, to study the gain from each component. From the best of these two settings (exclusive and undefined), we report the performance of (i) removing
the subtyping constraint as is described in subsection 4.5; (ii) substituting the multi-level margins in Equation 7 with a "flat" margin, i.e., margins on all levels are set to be 1. These results are shown in Table 2 and Table 3 under our best results, and they show that both multi-level margins and subtyping relation constraints offer orthogonal improvements to our models.
Error Analysis We identify common patterns of errors, coupled with typical examples:
- Confusing types: In BBN, our model outputs /gpe/city when the gold type is /location/region for "... in shipments from the Valley of either hardware or software goods." These types are semantically similar, and our model failed to discriminate between these types.
- Incomplete types: In FIGER, given instance "... multi-agency investigation headed by the U.S. Immigration and Customs Enforcement's homeland security investigations unit", the gold types are /government_agency and /organization, but our model failed to output /organization.
- Focusing on only parts of the mention: In AIDA, given instance "... suggested they were the work of Russian special forces assassins
8472
out to blacken the image of Kievs pro-Western authorities", our model outputs /org/government whereas the gold type is /per/militarypersonnel. Our model focused on the "Russian special forces" part, but ignored the "assassins" part. Better mention representation is required to correct this, possibly by introducing type-aware mention representation—we leave this as future work.
We proposed (i) a novel multi-level learning to rank loss function that operates on a type tree, and (ii) an accompanying coarse-to-fine decoder to fully embrace the ontological structure of the types for hierarchical entity typing. Our approach achieved state-of-the-art performance across various datasets, and made substantial improvement  $(4 - 8\%)$  upon strict accuracy.
Additionally, we advocate for careful investigation into partial type paths: their interpretation relies on how the data is annotated, and in turn, influences typing performance.
We thank our colleague Guanghui Qin and the anonymous reviewers for their insightful suggestions and comments. This research benefited from support by the JHU Human Language Technology Center of Excellence (HLTCOE), and DARPA AIDA. The U.S. Government is authorized to reproduce and distribute reprints for Governmental purposes. The views and conclusions contained in this publication are those of the authors and should not be interpreted as representing official policies or endorsements of DARPA or the U.S. Government.
Abhishek, Ashish Anand, and Amit Awekar. 2017. Fine-grained entity type classification by jointly learning representations and label embeddings. In Proceedings of the 15th Conference of the European Chapter of the Association for Computational Linguistics, EACL 2017, Valencia, Spain, April 3-7, 2017, Volume 1: Long Papers, pages 797-807.
Kurt D. Bollacker, Colin Evans, Praveen Paritosh, Tim Sturge, and Jamie Taylor. 2008. Freebase: a collaboratively created graph database for structuring human knowledge. In Proceedings of the ACM SIGMOD International Conference on Management of
Data, SIGMOD 2008, Vancouver, BC, Canada, June 10-12, 2008, pages 1247-1250.
Eunsol Choi, Omer Levy, Yejin Choi, and Luke Zettle-moyer. 2018. Ultra-fine entity typing. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics, ACL 2018, Melbourne, Australia, July 15-20, 2018, Volume 1: Long Papers, pages 87-96.
Hongliang Dai, Donghong Du, Xin Li, and Yangqiu Song. 2019. Improving fine-grained entity typing with entity linking. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 6209-6214, Hong Kong, China. Association for Computational Linguistics.
Luciano Del Corro, Abdalghani Abujabal, Rainer Gemulla, and Gerhard Weikum. 2015. FINET: context-aware fine-grained named entity typing. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, EMNLP 2015, Lisbon, Portugal, September 17-21, 2015, pages 868-878.
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACL-HLT 2019, Minneapolis, MN, USA, June 2-7, 2019, Volume 1 (Long and Short Papers), pages 4171-4186.
Matt Gardner, Joel Grus, Mark Neumann, Oyvind Tafjord, Pradeep Dasigi, Nelson F. Liu, Matthew Peters, Michael Schmitz, and Luke Zettlemoyer. 2018. AllenNLP: A deep semantic natural language processing platform. In Proceedings of Workshop for NLP Open Source Software (NLP-OSS), pages 1-6, Melbourne, Australia. Association for Computational Linguistics.
Dan Gillick, Nevena Lazic, Kuzman Ganchev, Jesse Kirchner, and David Huynh. 2014. Context-dependent fine-grained entity type tagging. CoRR, abs/1412.1820.
Nitish Gupta, Sameer Singh, and Dan Roth. 2017. Entity linking via joint encoding of types, descriptions, and context. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, EMNLP 2017, Copenhagen, Denmark, September 9-11, 2017, pages 2681-2690.
Xu Han, Shulin Cao, Xin Lv, Yankai Lin, Zhiyuan Liu, Maosong Sun, and Juanzi Li. 2018. OpenKE: An open toolkit for knowledge embedding. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, EMNLP 2018: System Demonstrations, Brussels, Belgium, October 31 - November 4, 2018, pages 139-144.
8473
Hailong Jin, Lei Hou, Juanzi Li, and Tiansi Dong. 2019. Fine-grained entity typing via hierarchical multi graph convolutional networks. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 4968-4977, Hong Kong, China. Association for Computational Linguistics.
Thorsten Joachims. 2002. Optimizing search engines using clickthrough data. In Proceedings of the Eighth ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, July 23-26, 2002, Edmonton, Alberta, Canada, pages 133-142.
Ying Lin and Heng Ji. 2019. An attentive fine-grained entity typing model with latent type representation. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 6198-6203, Hong Kong, China. Association for Computational Linguistics.
Xiao Ling and Daniel S. Weld. 2012. Fine-grained entity recognition. In Proceedings of the Twenty-Sixth AAAI Conference on Artificial Intelligence, July 22-26, 2012, Toronto, Ontario, Canada., pages 94-100.
Ilya Loshchilov and Frank Hutter. 2019. Decoupled weight decay regularization. In 7th International Conference on Learning Representations, ICLR 2019, New Orleans, LA, USA, May 6-9, 2019.
Minh-Thang Luong, Hieu Pham, and Christopher D. Manning. 2015. Effective approaches to attention-based neural machine translation. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, EMNLP 2015, Lisbon, Portugal, September 17-21, 2015, pages 1412-1421.
Shikhar Murty, Patrick Verga, Luke Vilnis, Irena Radovanovic, and Andrew McCallum. 2018. Hierarchical losses and new resources for fine-grained entity typing and linking. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics, ACL 2018, Melbourne, Australia, July 15-20, 2018, Volume 1: Long Papers, pages 97-109.
Yasumasa Onoe and Greg Durrett. 2019. Learning to denoise distantly-labeled data for entity typing. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACL-HLT 2019, Minneapolis, MN, USA, June 2-7, 2019, Volume 1 (Long and Short Papers), pages 2407-2417.
Matthew E. Peters, Mark Neumann, Mohit Iyyer, Matt Gardner, Christopher Clark, Kenton Lee, and Luke Zettlemoyer. 2018. Deep contextualized word representations. In Proceedings of the 2018 Conference of the North American Chapter of the Associ
ation for Computational Linguistics: Human Language Technologies, NAACL-HLT 2018, New Orleans, Louisiana, USA, June 1-6, 2018, Volume 1 (Long Papers), pages 2227-2237.
Jonathan Raiman and Olivier Raiman. 2018. Deep-type: Multilingual entity linking by neural type system evolution. In Proceedings of the Thirty-Second AAAI Conference on Artificial Intelligence, (AAAI-18), the 30th innovative Applications of Artificial Intelligence (IAAI-18), and the 8th AAAI Symposium on Educational Advances in Artificial Intelligence (EAAI-18), New Orleans, Louisiana, USA, February 2-7, 2018, pages 5406-5413.
Xiang Ren, Wenqi He, Meng Qu, Lifu Huang, Heng Ji, and Jiawei Han. 2016a. AFET: automatic fine-grained entity typing by hierarchical partial-label embedding. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, EMNLP 2016, Austin, Texas, USA, November 1-4, 2016, pages 1369-1378.
Xiang Ren, Wenqi He, Meng Qu, Clare R. Voss, Heng Ji, and Jiawei Han. 2016b. Label noise reduction in entity typing by heterogeneous partial-label embedding. In Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, San Francisco, CA, USA, August 13-17, 2016, pages 1825-1834.
Sonse Shimaoka, Pontus Stenetorp, Kentaro Inui, and Sebastian Riedel. 2016. An attentive neural architecture for fine-grained entity type classification. In Proceedings of the 5th Workshop on Automated Knowledge Base Construction, AKBC@NAACL-HLT 2016, San Diego, CA, USA, June 17, 2016, pages 69-74.
Sonse Shimaoka, Pontus Stenetorp, Kentaro Inui, and Sebastian Riedel. 2017. Neural architectures for fine-grained entity type classification. In Proceedings of the 15th Conference of the European Chapter of the Association for Computational Linguistics, EACL 2017, Valencia, Spain, April 3-7, 2017, Volume 1: Long Papers, pages 1271-1280.
Théo Trouillon, Johannes Welbl, Sebastian Riedel, Éric Gaussier, and Guillaume Bouchard. 2016. Complex embeddings for simple link prediction. In Proceedings of the 33nd International Conference on Machine Learning, ICML 2016, New York City, NY, USA, June 19-24, 2016, pages 2071-2080.
Ralph Weischedel and Ada Brunstein. 2005. BBN pronoun coreference and entity type corpus. *Philadelphia: Linguistic Data Consortium*.
Jason Weston and Chris Watkins. 1999. Support vector machines for multi-class pattern recognition. In *ESANN* 1999, 7th European Symposium on Artificial Neural Networks, Bruges, Belgium, April 21-23, 1999, Proceedings, pages 219-224.
8474
Wenhan Xiong, Jiawei Wu, Deren Lei, Mo Yu, Shiyu Chang, Xiaoxiao Guo, and William Yang Wang. 2019. Imposing label-relational inductive bias for extremely fine-grained entity typing. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACL-HLT 2019, Minneapolis, MN, USA, June 2-7, 2019, Volume 1 (Long and Short Papers), pages 773-784.
Peng Xu and Denilson Barbosa. 2018. Neural fine-grained entity type classification with hierarchy-aware loss. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACL-HLT 2018, New Orleans, Louisiana, USA, June 1-6, 2018, Volume 1 (Long Papers), pages 16-25.
Yadollah Yaghoobzadeh and Hinrich Schütze. 2015. Corpus-level fine-grained entity typing using contextual information. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, EMNLP 2015, Lisbon, Portugal, September 17-21, 2015, pages 715-725.
Dani Yogatama, Daniel Gillick, and Nevena Lazic. 2015. Embedding methods for fine grained entity type classification. In Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing of the Asian Federation of Natural Language Processing, ACL 2015, July 26-31, 2015, Beijing, China, Volume 2: Short Papers, pages 291-296.
Mohamed Amir Yosef, Sandro Bauer, Johannes Hoffart, Marc Spaniol, and Gerhard Weikum. 2012. HYENA: hierarchical type classification for entity names. In COLING 2012, 24th International Conference on Computational Linguistics, Proceedings of the Conference: Posters, 8-15 December 2012, Mumbai, India, pages 1361-1370.
Sheng Zhang, Kevin Duh, and Benjamin Van Durme. 2018. Fine-grained entity typing through increased discourse context and adaptive classification thresholds. In Proceedings of the Seventh Joint Conference on Lexical and Computational Semantics, *SEM@NAACL-HLT 2018, New Orleans, Louisiana, USA, June 5-6, 2018, pages 173-179.
8475