SlowGuess/ACL_2020_OCR huggingface dataset:
Data	Model	A@1	A@5	|C|	t
AIDA	MFLEB	0.56	0.71
LightGBM	0.44	0.72	31	9
RankSVM	0.37	0.69	56
RankNet	0.42	0.70	190
WWO	MFLEB	0.32	0.77
LightGBM	0.37	0.83	19	2
RankSVM	0.46	0.86	15
RankNet	0.52	0.87	37
1641	MFLEB	0.28	0.75
LightGBM	0.35	0.77	38	1
RankSVM	0.48	0.80	1
RankNet	0.55	0.83	2
Table 4: Ranking scores when using all the data. We report Accuracy@1 (Gold Candidate was ranked highest, Accuracy@5 (Gold Candidate was in top 5 predictions of the ranker)).  $|C|$  denotes the average number of candidates found for each mention. For AIDA, we evaluate on the test set, for the other datasets, we use 10-fold cross validation. We also measure the training time  $t$  in seconds averaged over 10 runs.
Feature importance The models we chose for ranking are white-box; they allow us to introspect the importance they give to each feature, thereby explaining their scoring choice. For the RankSVM, we follow Guyon et al. (2002) and use the square of the model weights as importance. For LightGBM, we use the number of times a feature is used to make a split in a decision tree. We train RankSVM and LightGBM models on all data and report the most important and least important features in Fig. 3. We normalize the weights by the L1-norm. It can be seen that both models rely on Levenshtein distance between mention and label as well as Sentence-BERT. The other text similarity features are, while sparingly, also used. Simple features like exact match, contains or prefix and postfix seem to not have a large impact. In general, LightGBM uses more features than the RankSVM. Even though Sentence-BERT was trained on Natural Language Inference (NLI) data which contains only relatively simple sentences, it still is relied on by both models for all datasets. The high importance of Levenshtein distance between mention and label for 1641 is expected and can be explained by the fact that the knowledge base labels often were derived from the mentions in the text when creating a domain-specific knowledge
6987
base for this dataset. When trained on AIDA, the RankSVM assigns a high importance to the Jaccard distance between context and description. We attribute this to the fact that entity descriptions in Wikidata are quite short; if they are similar to the context then it is very likely a match.
Figure 3: Feature importance of the respective models for different datasets. For the RankSVM, we use the squared weights; for LightGBM, we use the number of times a feature is used for splitting. Both are normalized to sum up to 1. ML stands for Mention-Label, CD for Context-Description.
5.3 Simulation
We simulate the Human-In-The-Loop setting by modeling a user annotating an unannotated corpus linearly. In the beginning, they annotate an initial seed of 10 entities without annotation support which are then used to bootstrap the ranker. At every step, the user annotates several entities where the ranker is used as assistance. After an annotation batch is finished, this new data is added to the training set, the ranker is retrained and evaluated. Only LightGBM and RankSVM are used as the RankNet turned out to be too slow. We do not evaluate on a holdout set. Instead, we follow Erdmann et al. (2019) and simulate annotating the complete corpus and evaluate on the very same data as we are interested in how an annotated sub
set helps to annotate the rest of the data, not how well the model generalizes. We assume that users annotate mention spans perfectly, i.e. we use gold spans. The candidate generation is simulated in three phases. It relies on the fact that the gold entity is given by the dataset: First, search for the mention only. If it was not found, search for the first word of the mention only. If this does not return the gold entity, search for the gold entity label. All candidates retrieved by these searches for a mention are used as training data. We also experimented with using only candidates for that the ranker assigned a higher score than the gold one. This, however, did not affect the performance. Therefore, we use all negative candidates.
Fig. 4 depicts the simulation results. All models outperform the MFLE baseline over most of the annotation process. It can be seen that both of our used models achieve high performance even if trained on very few annotations. The RankSVM handles low data better than LightGBM, but quickly reaches its peak performance due to it being a linear model with limited learning capacity. The LightGBM does not plateau that early. This potentially allows to first use a RankSVM for the cold start and when enough annotations are made, LightGBM, thereby combining the best of both models. Comparing the performance on the three datasets, we notice that the performance for AIDA is much higher. Also, the baseline rises much more steeply, hinting again that AIDA is easier and popularity there is a very strong feature. For 1641, the curve continues to rise, hinting that more data is needed to reach maximum performance.
Dataset	Phase 1	Phase 2	Phase 3
AIDA	0.20	0.00	0.80
WWO	0.26	0.27	0.47
1641	0.55	0.06	0.39
Table 5: Percentage of times the simulated user found the gold entity in the candidate list by searching for the mention (Phase 1), for the first word of the mention (Phase 2) or for the gold label (Phase 3).
Table 5 shows how the simulated user searched for the gold entities. We see that for WWO and 1641, the user often does not need to spend much effort in searching for the gold label, using the mention is in around  $50\%$  of the cases enough. We attribute this to the fuzzy search which the official Wikidata endpoint does not offer.
6988
Figure 4: Human-in-the-loop simulation results for our three datasets and models. We can see that we get good Accuracy@5 with only a few annotations, especially for the RankSVM. This shows that the system is useful even at the beginning of the annotation process, alleviating the cold start problem.
5.4 User Study
In order to validate the viability of our approach in a realistic scenario, we conduct a user study. For that, we augmented the already existing annotation tool INCEpTION $^5$  (Klie et al., 2018) with our Human-In-The-Loop entity ranking and automatic suggestions. Fig. 5 shows a screenshot of the annotation editor itself. We let five users reannote parts of the 1641 corpus. It was chosen as it has a high density of entity mentions while being small enough to be annotated in under one hour. Users stem from various academic backgrounds, e.g. natural language processing, computer science and digital humanities. Roughly half of them have previous experience with annotating. We compare two configurations: one uses our ranking and Levenshtein recommender, one uses the ranking of the full text search with the string matching recommender. We randomly selected eight documents which we split in two sets of four documents. To reduce bias, we assign users in four groups based on which part and which ranking they use first. Users are given detailed instructions and a warm-up document that is not used in the evaluation to get used to the annotation process. We measure annotation time, number of suggestions used and search queries performed. After the annotation is finished, we ask users to fill out a survey asking which system they prefer, how they experienced the annotation process and what suggestions they have to improve it. The evaluation of the user study
shows that using our approach, users on average annotated  $35\%$  faster and needed  $15\%$  less search queries. Users positively commented on the ranking performance and the annotation suggestions for both systems. For our ranking, users reported that the gold entity often ranked first or close to top; they rarely observed that gold candidates were sorted close to the end of the candidate list.
We conduct a paired sample t-test to estimate the significance of our user study. Our null-hypothesis is that the reranking system does not improve the average annotation time. Conducting the test yields the following:  $t = 3.332$ ,  $p = 0.029$ . We therefore reject the null hypothesis with  $p = 0.029 < 0.05$ , meaning that we have ample evidence that our reranking speeds up annotation time.
Recommender suggestions made up around  $30\%$  of annotations. We did not measure a significant difference between string and Levenshtein recommender. About the latter, users liked that it can suggest annotations for inexact matches. However, they criticized the noisier suggestions, especially for shorter mentions (e.g. annotating joabe (a name) yielded suggestions for to be). In the future, we will address this issue by filtering out more potentially unhelpful suggestions and using annotation rejections as a blacklist.
We presented a domain-agnostic annotation approach for annotating entity linking for low-resource domains. It consists of two main com
<sup>5</sup>https://inception-project.github.io
6989
Figure 5: For our user study, we extend the INCEpTION annotation framework:  $①$  entity linking search field,  $②$  candidate list,  $③$  linked named entity,  $④$  entity linking recommendation.
ponents: recommenders that are algorithms that suggest potential annotations to users and a ranker that, given a mention span, ranks potential entity candidates so that they show up higher in the candidate list, making it easier to find for users. Both systems are retrained whenever new annotations are made, forming the Human-In-The-Loop.
Our approach does not require the existence of external resources like labeled data, tools like named entity recognizers or large-scale resources like Wikipedia. It can be applied to any domain, only requiring a knowledge base whose entities have a label and a description. In this paper, we evaluate on three datasets: AIDA, which is often used to validate state-of-the-art entity linking systems as well as WwO and 1641 from the humanities. We show that in simulation, only a very small subset needs to be annotated (fewer than 100) for the ranker to reach high accuracy. In a user study, results show that users prefer our approach compared to the typical annotation process; annotation speed improves by around  $35\%$  when using our system relative to using no reranking support.
In the future, we want to investigate more powerful recommenders, combine interactive entity linking with knowledge base completion and use online learning to leverage deep models, despite their long training time.
We thank the anonymous reviewers and Kevin Stowe for their detailed and helpful comments. We also want to thank the Women Writers Project
which made the Women Writers Online text collection available to us. This work was supported by the German Research Foundation under grant No EC 503/1-1 and GU 798/21-1 as well as by the German Federal Ministry of Education and Research (BMBF) under the promotional reference 01UG1816B (CEDIFOR).
Sebastian Arnold, Robert Dziuba, and Alexander Löser. 2016. TASTY: Interactive Entity Linking As-You-Type. In Proceedings of COLING 2016, the 26th International Conference on Computational Linguistics: System Demonstrations, pages 111-115.
Sabine Bartsch. 2004. Annotating a Corpus for Building a Domain-specific Knowledge Base. In Proceedings of the Fourth International Conference on Language Resources and Evaluation (LREC'04), pages 1669-1672.
Carmen Brando, Francesca Frontini, and Jean-Gabriel Ganascia. 2016. REDEN: Named Entity Linking in Digital Literary Editions Using Linked Data Sets. Complex Systems Informatics and Modeling Quarterly, (7):60-80.
Chris Burges, Tal Shaked, Erin Renshaw, Ari Lazier, Matt Deeds, Nicole Hamilton, and Greg Hullender. 2005. Learning to rank using Gradient Descent. In Proceedings of the 22nd international conference on Machine learning - ICML '05, pages 89-96.
Diego Ceccarelli, Claudio Lucchese, Salvatore Orlando, Raffaele Perego, and Salvatore Trani. 2013. Dexter. In Proceedings of the sixth international workshop on Exploiting semantic annotations in information retrieval - ESAIR '13, pages 17-20.
Koby Crammer and Yoram Singer. 2003. Ultraconservative Online Algorithms for Multiclass Problems. JMLR, 3:951-991.
Joachim Daiber, Max Jakob, Chris Hokamp, and Pablo N. Mendes. 2013. Improving efficiency and accuracy in multilingual entity extraction. In Proceedings of the 9th International Conference on Semantic Systems - I-SEMANTICS '13, pages 121-124.
Yadollah Dodge. 2008. The Concise Encyclopedia of Statistics. Springer.
Alexander Erdmann, David Joseph Wrisley, Benjamin Allen, Christopher Brown, Sophie Cohen-Bodenes, Micha Elsner, Yukun Feng, Brian Joseph, Beatrice Joyeux-Prunel, and Marie-Catherine de Marneffe. 2019. Practical, Efficient, and Customizable Active Learning for Named Entity Recognition in the Digital Humanities. In Proceedings of the 2019 Conference of the North, pages 2223-2234.
Paolo Ferragina and Ugo Scaiella. 2010. TAGME: On-the-fly Annotation of Short Text Fragments (by Wikipedia Entities). In Proceedings of the 19th ACM international conference on Information and knowledge management - CIKM '10, pages 1625-1628.
Yang Gao, Christian M. Meyer, and Iryna Gurevych. 2018. APRIL: Interactively Learning to Summarise by Combining Active Preference Learning and Reinforcement Learning. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 4120-4130.
Stephen Guo, Ming-Wei Chang, and Emre Kiciman. 2013. To Link or Not to Link? A Study on End-to-End Tweet Entity Linking. In Proceedings of the 2013 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 1020–1030.
Isabelle Guyon, Jason Weston, Stephen Barnhill, and Vladimir Vapnik. 2002. Gene Selection for Cancer Classification using Support Vector Machines. Machine Learning, 46:389-422.
Luheng He, Julian Michael, Mike Lewis, and Luke Zettlemoyer. 2016. Human-in-the-Loop Parsing. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pages 2337-2342.
Johannes Hoffart, Mohamed Amir Yosef, Ilaria Bordino, Hagen Fürstenau, Manfred Pinkal, Marc Spaniol, Bilyana Taneva, Stefan Thater, and Gerhard Weikum. 2011. Robust Disambiguation of Named Entities in Text. In Proceedings of EMNLP'11, pages 782-792.
Filip Ilievski, Piek Vossen, and Stefan Schlobach. 2018. Systematic Study of Long Tail Phenomena in Entity Linking. In Proceedings of the 27th International Conference on Computational Linguistics, pages 664-674.
Thorsten Joachims. 2002. Optimizing search engines using clickthrough data. In Proceedings of the eighth ACM SIGKDD international conference on Knowledge discovery and data mining - KDD '02, pages 133-142.
Guolin Ke, Qi Meng, Thomas Finley, Taifeng Wang, Wei Chen, Weidong Ma, Qiwei Ye, and Tie-Yan Liu. 2017. LightGBM: A Highly Efficient Gradient Boosting Decision Tree. In I. Guyon, U. V. Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett, editors, Advances in Neural Information Processing Systems 30, pages 3146-3154.
Jan-Christoph Klie, Michael Bugert, Beto Boullosa, Richard Eckart de Castilho, and Iryna Gurevych. 2018. The INCEpTION Platform: Machine-Assisted and Knowledge-Oriented Interactive Annotation. In Proceedings of the 27th International Conference on Computational Linguistics: System Demonstrations, pages 5-9.
Edgar Meij, Krisztian Balog, and Daan Odijk. 2014. Entity linking and retrieval for semantic search. In Proceedings of the 7th ACM international conference on Web search and data mining - WSDM '14, pages 683-684.
John Melson and Julia Flanders. 2010. Not Just One of Your Holiday Games: Names and Name Encoding in the Women Writers Project Textbase. White paper, Women Writers Project, Brown University.
David Milne and Ian H. Witten. 2008. Learning to link with Wikipedia. In Proceeding of the 17th ACM conference on Information and knowledge mining - CIKM '08, pages 509-518.
Gary Munnelly and Séamus Lawless. 2018a. Constructing a knowledge base for entity linking on Irish cultural heritage collections. Procedia Computer Science, 137:199-210.
Gary Munnelly and Seamus Lawless. 2018b. Investigating Entity Linking in Early English Legal Documents. In Proceedings of the 18th ACM/IEEE on Joint Conference on Digital Libraries - JCDL '18, pages 59-68.
Farhad Nooralahzadeh and Lilja Øvrelid. 2018. SIRIUS-LTG: An Entity Linking Approach to Fact Extraction and Verification. In Proceedings of the First Workshop on Fact Extraction and VERIFICATION (FEVER), pages 119-123.
Nils Reimers and Iryna Gurevych. 2019. SentenceBERT: Sentence Embeddings using Siamese BERT-Networks. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing, pages 3980-3990.
Matthias Schlogl and Katalin Lejtovicz. 2017. APIS - Austrian Prosopographical Information System. In Proceedings of the Second Conference on Biographical Data in a Digital World 2017.
6991
Klaus U. Schulz and Stoyan Mihov. 2002. Fast string correction with Levenshtein automata. International Journal on Document Analysis and Recognition, 5(1):67-85.
Wei Shen, Jianyong Wang, and Jiawei Han. 2015. Entity Linking with a Knowledge Base: Issues, Techniques, and Solutions. IEEE Transactions on Knowledge and Data Engineering, 27(2):443-460.
Ricardo Usbeck, Axel-Cyrille Ngonga Ngomo, Michael Röder, Daniel Gerber, Sandro Athaide Coelho, Soren Auer, and Andreas Both. 2014. AGDISTIS - Graph-Based Disambiguation of Named Entities Using Linked Data. In *The Semantic Web - ISWC* 2014, pages 457-471.
Eric Wallace, Pedro Rodriguez, Shi Feng, Ikuya Yamada, and Jordan Boyd-Graber. 2019. Trick Me If You Can: Human-in-the-loop Generation of Adversarial Question Answering Examples. Transactions of the Association for Computational Linguistics, 7(0):387-401.
Travis Wolfe, Mark Dredze, James Mayfield, Paul McNamee, Craig Harman, Tim Finin, and Benjamin Van Durme. 2015. Interactive Knowledge Base Population.
Seid Muhie Yimam, Chris Biemann, Richard Eckart de Castilho, and Iryna Gurevych. 2014. Automatic Annotation Suggestions and Custom Annotation Layers in WebAnno. In Proceedings of 52nd Annual Meeting of the Association for Computational Linguistics: System Demonstrations, pages 91-96.
Zhicheng Zheng, Fangtao Li, Minlie Huang, and Xiaoyan Zhu. 2010. Learning to Link Entities with Knowledge Base. In Proceedings of NAACL-HLT'10, pages 483-491.
A.1 Dataset creation
The following section describes how we preprocess the raw texts from wwo and 1641. Example texts can be found in Table 6. The respective code and datasets will be made available on acceptance.
A.1.1 Women Writers Online
We use the following checkout of the WWO data, which was graciously provided by the Women Writers Project<sup>6</sup>.
Revision: 36425
Last Changed Rev: 36341
Last Changed Date: 2019-02-19
<sup>6</sup>https://www.wwp.northeastern.edu/
The texts itself are provided as TEI<sup>7</sup>. We use DKPro Core<sup>8</sup> to read in the TEI, split the raw text into sentences and tokenize it with the J Tok Segmenter. When an annotation is spread over two sentences, we merge these sentences. This is mostly caused by a too eager sentence splitter. We covert the personographie which is in XML to RDF, including all properties that were encoded in there.
A.1.2 1641 Depositions
We use a subset of the 1641 depositions provided by Gary Munnelly. The raw data can be found on  $\mathrm{Gibdb}^9$ . The texts itself are provided as  $\mathrm{NIF}^{10}$ . We use DKPro Core<sup>11</sup> to read in the NIF, split the raw text into sentences and tokenize it with the JTokSegmenter. When an annotation is spread over two sentences, we merge these sentences. This is mostly caused by a too eager sentence splitter. We use the knowledge base that comes with the NIF and create entities for all mentions that were NIL. We carefully deduplicate entities, e.g. Luke Toole and Colonel Toole are mapped to the same entity. In order to increase the difficulty of this dataset, we add additional entities from DBPedia: all Irish people, Irish cities and buildings in Ireland; all popes; royalties born between 1550 and 1650.
For that, we execute SPARQL queries against DBPedia for instances of dbc:Popes, dbc:Royalty, dbc:17th-century_Irish_people and keep entries with a birth date before 1650 and a death date between 1600 and 1700. For the places, we search for dbo:Castle, dbo:HistoricPlace, dbo:Building, dbc:17th-century_Irish_people that are located in Ireland. The following table shows how many entities were in the original KB and how many were added:
Persons in gold data 130
Places in gold data 99
Persons added from DBPedia 1253
Places added from DBPedia 2020
<sup>7</sup>https://tei-c.org/
<sup>8</sup>https://dkpro.github.io/dkpro-core/
<sup>9</sup>https://github.com/munnellg/
1641DepositionsCorpus
10https://persistence.uni-leipzig.org/nlp2rdf/
https://dkpro.github.io/dkpro-core/
6992
WWO
The following Lines occasion'd by the Marriage of Edward Herbert Esquire, and Mrs. Elizabeth Herbert. Cupid one day ask'd his Mother, When she meant that he shou'd Wed? You're too Young, my Boy, she said: Nor has Nature made another Fit to match with Cupid's Bed.
Finch, Anne: Miscellany poems, on several occasions, 1713
Joseph Joice of Kisnebrasney in the kings County gentleman sworne and examined deposeth and saith That after the Rebellion was begun in the County aforesaid vizt about the xxth of November 1641 This deponent for safty fled to the Castle of knocknamease in the same County
Deposition of Joseph Joice,  $1643^{12}$
Table 6: Example sentences from these corpora. Linked Named entities are highlighted in yellow.
A.2 Experiments
A.2.1 Full text search
For AIDA and Wikidata, we use the official SPARQL endpoint and the Mediawiki API Query Service<sup>13</sup>. It does not support fuzzy search. For wwo and 1641, we host the created RDF in a Fuseki<sup>14</sup> instance and use the builtin functionality to index via Lucene.
A.2.2 Timing
Timing was performed on a Desktop PC with Ryzen 3600 and a GeForce RTX 2060.
$^{13}$ https://www.mediawiki.org/wiki/Wikidata 查询/Service/User_Manual/MWAPI $^{14}$ https://jena.apache.org/documentation/fuseki2/
6993