mReFinED: An Efficient End-to-End Multilingual Entity Linking System
Authors: Peerat Limkonchotiwat
Weiwei Cheng
Christos Christodoulopoulos
Amir Saffari
Jens Lehmann
Venue: EMNLP 2023 Findings (2023)
Keywords: entity linking
multilingual
end-to-end

Abstract: End-to-end multilingual entity linking (MEL) is concerned with identifying multilingual entity mentions and their corresponding entity IDs in a knowledge base. Existing works assumed that entity mentions were given and skipped the entity mention detection step due to a lack of high-quality multilingual training corpora. To overcome this limitation, we propose mReFinED, the first end-to-end multilingual entity linking. Additionally, we propose a bootstrapping mention detection framework that enhances the quality of training corpora. Our experimental results demonstrated that mReFinED outperformed the best existing work in the end-to-end MEL task while being 44 times faster.

mReFinED:
An Efficient End-to-End Multilingual Entity Linking System
Peerat Limkonchotiwat'; Weiwei Cheng", Christos Christodoulopoulos,
Amir Saffari', Jens Lehmann²
School of Information Science and Technology, VISTEC, Thailand
2 Amazon
1eerat.1_19@vvisteo ac
(wwcheng, chrchrs, amsafari, jlehmnn @amazon. com
Abstract
models (Devlin al., 2019). An alternative method predicting the target entity's Wikipedia title in an auto-regressive manner (Cao et al., 2022) by extending the sequence-to-segeee sequence pipeline of Cao et al. (2021). However, none of the existing works perform end-to-end MEL because of a lack of high-quality multilingual entity mention training resources. For example, found that Wikipedia suffers from an unlabelled entity mention problem, i.e. all entity mentions have hyperlink markups to train a reliable mention detection model. Thus, devising an end-to-end MEL system remains a challenging task. In this paper, we propose the first end-to-end MEL system. To address the unlabelled mention problem in end-to-end MEL,, we propose bootstrapping mention detection (MD) framework. Our framework leverages an existing multilingual MD model to create bootstrapped dataset, which we use to train new mention detection model for annotating unlabelled mentions in Wikipedia. The framework provides an improvement for detecting named and non-named entities in Wikipedia, compared to previous multilingual MD approaches (Honnibal et al.. 2020; Hu et al., 2020; Tedeschi et al., 2021). To construct the end-to-end MEL system, extend ReFinED (Ayoola et al., 2022) since comparable to the -fate-of-the-art (SOTA) models the English end-to-end EL setting, and significantly faster than any other methods to date. We call this new model mRe- FinED. Our code is released at: https: wwtp://.oo///c/occc.c... ReF ReFinDD/tree//eeffinee
End-to-end multilingual entity linking (MEL)
is concerned with identifying multilingual en- tity mentions and their corresponding entity IDs a knowledge base. Prior efforts assume that entity mentions given and skip the tity mention detection step due to lack of high- quality multilingual training corpora. To over- this limitation, propose mReFinED, the first end-to-end MEL model. Additionally, we propose bootstrapping mention detection framework that enhances the quality of train- ing corpora. Our experimental results demon- strated that mReFinED outperformed the best existing work in the end-to-end MEL task while being 44 times faster.
1 Introduction
End-to-end entity linking (EL) is the task of identi- fying entity mentions within a given text and map- ping them to the corresponding entity in knowl- edge base. End-to-end EL plays crucial role in various NLP tasks, such as question answering (Nie et al., 2019; Asai et al., 2020; Hu et al., 2022) and information retrieval (Zhang et al., 2022).
To clarify the terminology used in this paper and previous work, it should be noted that when referring to EL in previous work, we are specifically referring to entity disambiguation (ED) where the entity mentions are given. We will only refer to our definition of EL (mention detection with ED) using the full name "end-to-end EL". Existing EL research has extended models to a single model
support over 100 languages using Wikipedia as the training corpus. We call this task multilingual entity linking (MEL). Recent work proposed MEL frameworks by minimizing discrepancy between mention and entity descrip- tion representation (Botha et al., 2020) the same mention but in different contexts (FitzGerald et al., 2021) based on bi-encoder pre-trained language
To demonstrate mReFinED's effectiveness, we compare it with SOTA MEL (Cao et al., 2022) the end-to-end MEL task across two datasets. Mewsli-9 (Botha al., 2020) and TR2016 (Tsai and Roth, 2016). Experimental results show that mReFinED outperforms a two-stage model (com- bining SOTA MD and MEL models) on both datasets. Moreover, mReFinED's inference speed
Work conducted during Research Internship at Amazon.

---

is 44 times faster than SOTA MEL.
tuning (using the BIO tagging format; Ramshaw and Marcus, 1995). When there are possibilities of overlapping entities, we only use the longest entity mention. In addition, we found that using only 200k sentences per language as training data is enough for formulating the bootstrapped MD (see Appendix A.3 for further details). Finally, we use the new MD model to annotate unlabelled mentions in Wikipedia. The main advantage of using the new MD model over the XTREME model its ability to detect both named and common noun entities. This is because the new MD model learned named entities from the XTREME model's annotation and common noun enttties from existing markups in Wikipedia, which is useful since most entities in current MEL datasets are common noun entities. However, the XTREME model is trained only on NER datasets and using it on MEL datasets would harm performance.
Our contributions are as follows: We propose the first end-to-end MEL a single model by extend- ing ReFinED to multilingual ReFinED. In addi- tion, we propose a bootstrapping mention detection framework to solve the unlabelled mention prob- lem in end-to-end MEL.
2 Methodology
Overview. We first fine-tune a mention detection (MD) model based on a multilingual pre-trained language model (PLM) - with our bootstrapping MD framework, as shown in Figure 1. We then use the bootstrapped MD model to annotate unla- belled mentions in Wikipedia. Finally, we use the data bootstrapped to train mReFinED in multi- task manner (Ayoola et al., 2022), which includes mention detection, entity type prediction, entity description, and entity disambiguation.
2.2 Task Definition: end-to-end MEL
2.1 Bootstrapping Mention Detection
Given a sequence of words in the document W = {wi, W22 .... w]w]), where the document can be written in multiple languages. We iden- tify entity mentions within the document M = [m1, m.. and map each one to the cor- responding entity E = (e1, ... C in knowl- edge base (KB).
Bootstrapping mention detection (MD) framework
Multilingual
Existing hyperlinks
Roatstrapping
NER
annotations ground truths
MD model
Adding
Fine-tuning
mnnotation
MD process
Augmented datasets
Multilingual Wikipedia
Multilingual BERT
2.3 Mention Detection
Adding ammotstion to training data Using the new MD model to add new ancutations
We use mBERT as our mention encoder to encode the words wy with the output from the last layer serving as the contextualized word representations h; We add a linear layer to our mention encoder to train token classification (BIO tagging format) from h, using cross-entropy LMD to the gold label. Then, we obtain mention representations m; for each my by average pooling h; of the entity men- tion tokens. All words W are encoded in single forward pass, resulting in fast inference.
Randomly pick only Z00R
sontences/aangaage
Multilingual
Wikipedia
Figure I: The bootstrapping MD framework's overview.
As shown in Figure 1, we employ an ex- isting multilingual named entity recognition (NER) model to annotate unlabelled mentions in Wikipedia corpora. Our framework allows various choices of existing NER models (Honnibal al., 2020; Hu et al., 2020; Tedeschi et al., 2021) without any constraints. Based on our MD experiments in section 3.3, we adopted the XTREME NER model (fine-tuned on 40 languages) since it supports every language in both MEL datasets.
2.4 Entity Typing Score
Previous work in English EL (Raiman and Raiman, 2018; Onoe and Durrett, 2020) has shown that us- ing an entity typing model to link entities in KB can improve the accuracy of the EL task. Thus, we train fine- fine-grained entity typing model to predict entity types / of each mention m; where is set of entity types & E T from the KB. We add a linear layer fo, with sigmoid activation a map m; to a fixed-size vector. We then calculate the entity typing score ET(e,, m²) using Euclidean distance
We then train bootstrapped multilingual MD model using the same PLM as previous MD works (mBERT; Devlin et al., 2019) with the newly anno- tated mentions and existing markups in Wikipedia. For simplicity, we train the bootstrapped MD model similarly to BERT's token classification fine-

---

between (Je,) and multi-label entity types T': ET(ej,m)) (fo (mi)) - T (I)
my and dj with additional temperature scaling (similarity score calibration; Guo et al., 2017); and (iii) entity prior score (P(e)m)) derived from Wikipedia hyperlink count statistics and Wikidata aliases (Hoffart et al., 201 1). These three outputs are passed through linear layer fe, with an output dimension of one, shown below:
We formulate T by assigning value of one to the correct entity types T and a value of zero to the rest (one entity can have multiple types). We then minimize the distance between the gold label (T") and E.( using cross-entropy LET following the distantly type labels from Onoe and Durrett (2020).
EL fe, (ET(e,, mi);simmmm))) sim (()/T;P(e))m))
(3)
The output from EL is a score for each Ej corre- sponding to m,- We train the EL score by mínimiz- ing the difference between EL and the gold label using cross-entropy LEI.
2.5 Entity Description Score
In this module, we compute the cross-lingual sim- ilarity score between the mention my and entity description (dj) in the KB. We use English as the primary language of dj since English is the domi- nant language for mBERT (the language with the highest amount of data), and the model tends to perform substantially better compared to other lan- guages (Arivazhagan et al., 2019; Limkonchotiwat et al., 2022). When English is unavailable we ran- domly select another language. We use another BBER to encode entity descriptions dj and train jointly with our mention encoder. We formulate dj as [CLS] label [SEP] description [SEP] and derive the contextualized representation dj from the [CLS] token in the final layer embedding. We incorporate linear layers foz and fe, to mention and description encoders, respectively, with L2- normalization the output of linear layers.
2.7 Multi-tast training
We train mReFinED in multi-task manner by combining all losses in single forward pass.
ADD + A2fET + A3CED + dA (4)
During the training process, we use provide en- tity mentions (hyperlink markups), and we simul- tancously train MD (hyperlink markups and new annotations in Section 2.1) along with other tasks.
Experiments
3.1 Experiment Setting
Sctup. We used the Wikipedia data and articles from 11 languages with timestamp of 20221203 as our training data. To generate candidates, we used the top-30 candidates from Ayoola et al. (2022) and concatenated them with the top-30 can- didates from Cao et al. (2022). We then select only the top 30 candidates with the highest entity prior scores for both training and inference steps. For the full parameter, language, and candidate generation settings, please refer to Appendix A. and A.2. Metric. We evaluate mReFinED on the end-to- end EL task in both MEL datasets (Mewsli-9 and TR2016 using the same metric in pre- vious MEL works, which is based on the recall score (Botha et al., 2020; FitzGerald et al., 2021; Cao et al., 2022).
Prior works used cosine similarity derive de- scription score for each entity (Botha et al., 2020; FitzGerald et al., 2021; Ayoola et al., 2022). In con- trast, we employ NT-Xent (the temperature-scaled cross-entropy loss) as our training objective since it demonstrated better robustness in ranking results compared to cosine similarity (Chen et al., 2020):
Simm for fog (dj))/r
CED -log
(2)
Σ EdED sim(fa (m:).Ja, (d))/r
where D denotes set of description produced by candidate generation, denotes the temperature parameter, and sim(-) denotes the cosine similarity between two feature vectors.
3.2 End-to-End MEL Results
2.6 Entity Disambiguation Score
Table 1 presents the performance of mReFinED and mGENRE both MEL datasets. The performance of mReFinED is compared with that of two- stage model, which involves bootstrapping MD with mGENRE. Our experiment results on Mewsli- demonstrated that mReFinED outperforms the
Prior studies demonstrated that using entity descrip- tion benefits entity disambiguation (Logeswaran et al., 2019; Wu al., 2020). Therefore, we concatenate three outputs: (i) entity typing score (ET); (ii) the lingull similarity score between

---

Mewsli-9
TR2016 hard
Model
or
en
65
Ju
SE
ta
tr
macro
micro
es
macro
MD+-GENREE
59.0
67.9
62.1
67.5
54.0
38.3
83.7
34.8
46.4
57.1
63.6
30.5
31.5
23.4
25.5
27.7
mReFinED
61.8
69.3
64.2
68.0
54.2
43.5
84.5
33.7
49.8
58.8
65.5
28.2
34.4
25.3
25.8
28.4
w/o entity priors
60.5 62.5
61.3
63.5
54,5
42.8
83.0
33.2
47.2
56.5
61.9
29.1
34.2
25.8
26.0
28.8
w/o entity types
61.8 69.4
64.2
67.7
54.1
43.1
84.4
33.6
49.8
58.7
65.4
28.0
32.7
24.5
25.5
27.7
w/o descriptions
61.8
69.9
63.0
67.2
53.2
42.4
84.0
33.5
50.1
58.3
65.0
8.2
17.8
9.6
11.3
w/o bootstrapping
0.0
0.0
0.0
0.0
0.0
0.0
0.0
0.0
0.0
0.0
0.0
Table 1: Recall on Mewsli-9 and TR2016 datasets. We report both datasets' results of entity disambignation and entity linking tasks. We use the bootstrapping MD model mention detection for mGENRE.
two-stage model on the micro- and macro-averages by 1.8 and 1.9 points, respectively. The experi- mental results from 6R201 also demonstrated the same results as Mewsli-9. These results high- light the essentials of the end-to-end system in single model, which is better than the two-stage model. For precision and Fl scores, see table in Appendix. The ablations in Table show that, when the bootstrapping MD framework is removed and using only the Wikipedia markups, mReFinED produces zero scores for almost all languages in both datasets. This is because the number of entity mentions in the training data decreased from 880 million only 180 million mentions. These results emphasize the importance of our bootstrapping MD framework, which can effectively mitigate the un- labelled entity mention problem in Wikipedia.
leave further investigations for future work. On TR2016hard entity descriptions show the most contribution, +17.1 macro average recall when added. Entity types show a small amount of contribution, with +0.7 macro average recall, Entity priors turn out to be harmful when added except for es language. Macro average recall is +0.4 when entity priors removed. This could be explained by how mentions in TR201 dataset are selected. Mentions in TR2016 hhand dataset are chosen so that the correct entity did not appear as the top-ranked candidate by alias table lookup. This means entity priors are not very useful for finding the correct entity for these mentions and model needs to use other information such as en- tity descriptions and types to choose the correct entity. We believe this introduces discrepancy the training scenario where entity priors are very useful signal for finding the correct entity given a mention surface form. On the other hand, since the gap between with and without entity priors is small, it also demonstrates mReFinED model's ability to use appropriate information when entity priors alone is not enough to make correct predictions.
On Mewsli-9, entity priors and descriptions are slightly complementary and contribute +0.5 macro and micro average recall when combined. Entity types are less useful and contribute +0.1 on macro and micro average recall when added. Combining all three achieves the best macro and micro average recall 58.8 and 65.5 respectively.
For Arabic (ar), removing either entity types or descriptions has no difference and achieves the same recall (61.8) compared to using all informa- tion. This suggests entity types and descriptions are redundant when either one of them com- bined with entity priors. Entity priors hurt the performance in Farsi/Persian (fa) as removing gives +0.3 recall. However, there are only 535 Farsi mentions in the Mewsli-9 dataset, which is too small of sample size to draw reliable conclu- sions. For German (de) and Turkish (tr), remov- ing descriptions seems to be beneficial and yields recall gains of +0.6 and +0.3 respectively. This could be resource-specific issue (there could be fewer/lower quality Wikidata descriptions these two languages) or a language-relater issue (both languages are morphologically rich) but we will
3.3 Multilingual Mention Detection Results
This experiment compares the performance of mention detection models with prior multilin- gual MD works, such as spaCy (Honnibal et al., 2020), XTREME (Hu et al., 2020), and WikiNEu- Ral (Tedeschi et al., 2021) on both MEL datasets. We use the exact match score to evaluate the efficiency of this study following previous MD works (Tjong Kim Sang and De Meulder, 2003; Tsai et al., 2006; Diab et al., 2013). As shown in Table 9, our bootstrapping MD outperforms competitive methods in all languages; e.g., our bootstrapping MD outperformed XTREME by 9.9 points and 7.3 points on Mewsli-9 and TR2016hard datasets, respectively. In addition, mReFinED showed superior performance to the bootstrapping

---

MD by an average of 2.8 points on Mewsli-9, These results highlight the benefits from additional join training MD with other tasks a single model outperformed a single task model. We also run an experiment on XTREME NER dataset to hetter un- derstand our bootstrapping MD's performance on multilingual mention detection task. We expect our bootstrapping MD to achieve comparable results competitive multilingual NER models in the lit- erature when trained on NER data, please refer to Appendix A.5 for more details.
This presents an opportunity for us re-annotate the MEL datasets in the future using mReFinED an annotator tool to detect unlabelled mentions. Run-time Efficiency. This study measures the time per query of mReFinED compared to mGENRE on one 16 GB V100. Our findings in- dicate that mGENRE takes 1,280 ms + 36.1 ms to finish single query. In contrast, mReFinED requires only 29 ms 1.3 ms making 44 times faster than mGENRE because mReFinED encodes all mentions in a single forward pass.
3.4 Analysis
Conclusion
Incorrect labels in MEL datasets. It is notewor- thy that both Mewsli-9 and TR2016 datasets con- tains incorrect labels. In particular, we identified entities that were erroneously linked to the "Disam- biguation page" instead of their actual pages; c.g., the mention "imagine" in Mewsli-9 was linked to Q225777 "Wikimedia disambiguation page". Therefore, removed those incorrect labels in both datasets and re-evaluated mReFinEDD and mGENRE on the cleaned dataset in Table 2. mRe- FinED's performance on the cleaned Mewsli-9 dataset increases from 65.5 to 67.4 micro-avg, and mGENRE's performance increases from 63.6 to 65.7. Lastly, the number of entity mention in Mewsli-9 was decreased from 289, ,0,7 to 279,423 mentions. See Appendix A.4 for TR2016 results.
In this paper, we propose mReFinED, the first mul- tilingual end-to-end EL.. We extend the monolin- gual ReFinED to multilingual and add the new bootstrapping MD framework to mitigate the unla- belled mention problem. mReFinED outperformed SOTA MEL in the end-to-end EL task, and the inference speed is faster than SOTA 44 times.
Limitations
We did not compare mReFinED with other MEL works (Botha et al., 2020; FitzGerald et al., 2021) since they did not release their code. However, the experimental results from other MEL works in the ED task demonstrated lower performance than mGENRE. Our experiment report results are based on standard MEL datasets, such as Mewsli-9 and TR2016hard which may not reflect mReFinED's performance in real-world applications.
Cleanenl Mevali-
Method
MDHHGEENR
60.3
69.9
10.0
54.4
98.6
85.4
34,9
87.6
58.5
65.7
#RRFinED
63.0
71.3
66.8
70.1
54.6
44.0
86.3
34.0
50.9
68.1
67.4
Table 2: Recall score the cleaned Mewsli-9 dataset.
References
Unlabelled entity mentions in MEL datasets. It is important to note that the unlabelled entity men- tion problem also occurs in both MEL datasets. As mentioned in Section 2.3, most of entities in MEL datasets are common noun because these datasets use Wikipedia markups as entity mention ground truths. Thus, the MEL datasets also suffer from the unlabelled entity mention problem. For exam- ple, consider the document en-106602 in Mewsli-9 (Figure 3), it was annotated with only eight en- tity mentions, but mReFinED found an additional 1l entity mentions in the document, including lo- cation (i.e., "Mexico"), person (i.e., "Richard A. Feely"), organization (i.e.. "NOAA"), and com- mon noun (i.e., "marine algae") mentions. These results demonstrate that mReFinED can also miti- gate the unlabelled entity mention in MEL datasets.
Naveen Arivazhagan, Ankur Bapna, Orhan Firat, Dmitry Lepikhin, Melvin Johnson, Maxim Krikun, Mia Xu Chen. Yuan Cao, George Foster, Colin Cherry, Wolfgang Macherey, Zhifeng Chen, and Yonghui Wu. 2019. Massively multilingual neural machine translation in the wild: Findings and chal- lenges.
Akari Asai, Kazuma Hashimoto, Hannaneh Hajishirzi, Richard Socher, and Caiming Xiong. 2020. Learning to retrieve reasoning paths over wikipedia graph for question answering. In International Conference on Learning Representations.
Tom Ayoola, Shubhi Tyagi, Joseph Fisher, Christos Christodoulopoulos, and Andrea Pierleoni. 2022, Re- FinED: An efficient zero-shot-capable approach to end-to-end entity linking. In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human

---

Language Technologies: Industry Track, pages 209- 220, Hybrid: Seattle, Washington + Online. Associa- tion for Computational Linguisties.
Johannes Hoffart, Mohamed Amir Yosef, Ilaria Bordino, Hagen Fürstenau, Manfred Pinkal, Mare Spaniol, Bilyana Taneva, Stefan Thater, and Gerhard Weikum. 2011. Robust disambiguation of named entities in text. In Proceedings of the 2011 Conference on Em- pirical Methods in Natural Language Processing, pages 782-792, Edinburgh, Scotland, UK. Associa- tion for Computational Linguistics.
Jan A. Botha, Zifei Shan, and Daniel Gillick. 2020. En- tity Linking in 100 Languages. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 7833-7845, Online. Association for Computational Linguistics.
Matthew Honmibal, Ines Montani, Sofie Van Lan- deghem, and Adriane Boyd. 2020. spacy: Industrial- strength natural language processing in python.
Nicola De Cao, Gautier Izacard, Sebastian Riedel, and Fabio Petroni. 2021. Autoregressive entity retrieval. In 9th International Conference Learning Repre- sentations, ICLR 2021, Virtual Event, Austria, May 3-7, 2021. OpenReview.net.
Junjie Hu, Sebastian Ruder, Aditya Siddhant, Gra- ham Neubig, Orhan Firat, and Melvin Johnson, 2020. XTREME: A massively multilingual multi- task benchmark for evaluating cross-lingual gener- alisation. In Proceedings of the 37th International Conference on Machine Learning, volume 119 of Proceedings of Machine Learning Research, pages 4411 1411-4421. PMLR.
Nicola De Cao, Ledell Wu, Kashyap Popat, Mikel Artetxe, Naman Goyal, Mikhail Plekhanov, Luke Zettlemoyer, Nicola Cancedda, Sebastian Riedel, and Fabio Petroni. 2022. Multilingual autoregressive entity linking. Trans. Assoc. Comput. Linguistics, 10:274-290.
Xixin Hu, Xuan Wu, Yiheng Shu, and Yuzhong Qu. 2022. Logical form generation via multi-task learn- ing for complex question answering over knowledge bases. In Proceedings of the 29th International Con- ference on Computational Linguisties, pages 1687- 1696, Gyeongju, Republic of Korea. International Committee on Computational Linguistics.
Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. 2020. A simple framework for contrastive learning of visual representations. In Proceedings of the 37th International Conference on Machine Learning, volume 1 119 of Proceedings of Machine Learning Research, pages 1597-1607.
PMLR.
Peerat Limkonchotiwat, Wuttikorn Ponwitayarat, Can Udomcharoenchaikit, Ekapol Chuangsuwanich, and Sarana Nutanong, 2022. CL-ReLKT: Cross-lingual language knowledge transfer for multilingual re- trieval question answering. In Findings of the Associ- ation for Computational Linguistics: NAACL 2022, pages 2 2141-2155, Seattle, United States, Association for Computational Linguistics.
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language under- standing. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Tech- nologies, Volume I (Long and Short Papers). pages 4171-4186, Minncapolis, Minnesota. Association for Computational Linguistics.
Lajanugen Logeswaran, Ming-Wor Wei Chang, Kenton Lee, Kristina Toutanova, Jacob Devlin, and Honglak Lec. 2019. Zero-shot entity linking by reading entity de- scriptions. In Proceedings of the 57th Annual Meet- ing of the Association for Computational Linguistics. pages 3449-3460, Florence, Italy. Association for Computational Linguistics.
Mona T. Diab, Timothy Baldwin, and Marco Ba- roni, editors. 2013. Proceedings of the 7th International Workshop on Semantic Evaluation, SemEval @ NAACL-HLT 2013, Atlanta, Georgia, USA, June 14-15, 2013. The Association for Com- puter Linguisties.
Yixin Nie, Songhe Wang, and Mohit Bansal. 2019. Revealing the importance of semantic retrieval for machine reading at scale. In Proceedings of the 2019 Conference on Empirical Methods in Natu- rul Language Processing and the 9th International Joint Conference on Natural Language Processing DEMNLP-IJCNLL pages 2553-2566, Hong Kong. China. Association for Computational Linguistics.
Nicholas FitzGerald, Dan Bikel, Jan Botha, Daniel Gillick, Tom Kwiatkowski, and Andrew McCallum. 2021. MOLEMAN: Mention-only linking of entities with a mention annotation network. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 2: Short Papers), pages 278 285. Online. Association for Computational Linguistics.
Yasumasa Onoe and Greg Durrett. 2020. Fine-graine grained entity typing for domain independent entity linking. In The Thirty-Fourrhh AAAI Conference on Artificial Intelligence, AAAI 2020. The Thirty-Second Innova- tive Applications of Artificial Intelligence Conference, IAAI 2020, The Tenth AAAI Symposium on Educa- tional Advances in Artificial Intelligence, EAAI 2020, New York. NY, USA. February 7-12, 2020, pages 8576-8583. AAAI Press.
Chuan Guo, Geoff Pleiss, Yu Sun, and Kilian Q. Wein- berger. 2017. On calibration of modern neural net- works. In Proceedings of the 34th International Con- ference on Machine Learning, volume 70 of Pro- ceedings of Machine Learning Research, pages 1321- 1330. PMLR.

---

Jonathan Raiman and Olivier Raiman. 2018. Deeptype: Multilingual entity linking by neural type system evolution. In Proceedings of the Thirty-Second AAAI Conference on Artificial Intelligence, (AAAI-18), the 30th innovative Applications of Artificial Intelligence (IAAT-18), and the 8th AAAI Symposium Educa- tional Advances in Artificial Intelligence (FAAI-18) New Orleans, Louisiana, USA. February 2-7, 2018, pages 1400654113 AAAI Press.
Lance Ramshaw and Mitch Marcus. 1995. Text chunk- ing using transformation- based learning. In Third Workshop on Very Large Corpora.
Simone Tedeschi, Valentino Maiorca, Niccolò Campol- ungo, Francesco Cecconi, and Roberto Navigli, 2021, WikiNEuRal: Combined neural and knowledge- based silver data creation for multilingual NER. In Findings of the Association for Computational Lin- guisties: EMNLP 2021, pages 2521-2533, Punta Cana, Dominican Republic. Association for Compu- tational Linguisties.
Erik F. Tjong Kim Sang and Fien De Meulder. 2003. Introduction the CoNLL--2003 shared task: named entity recognition. In Proceedings of the Seventh Conference Natural Language Learning HLT-NAACL 2003, pages 142- 147.
Chen-Tss Tse Tsai and Dan Roth. 2016. Cross-lingual wiki- fication using multilingual embeddings. In Proceed- ings of the 2016 Conference of the North American Chapter of the Association for Computational Lin- guistics: Human Language Technologies, pages 589- 598. San Diego, California. Association for Compu- tational Linguistics.
Richard Tzong-Han Han Tsai, Shih-Hung Wu, Wen-Chi Chou, Yu-Chun Lin, Ding He, Jich Hsiang, Ting- Sung, and Wen-Lian Hsu. 2006. Various criteria in the evaluation of biomedical named entity recogni- tion. BMC Bioinform. 7:92.
Ledell Wu, Fabio Petroni, Martin Josifoski, Sebastian Riedel, and Luke Zettlemoyer. 2020. Scalable zero- shot entity linking with dense entity retrieval. In Proceedings of the 2020 Conference Empirical Methods in Natural Language Processing (EMNLP), pages 6397-6407, Online. Association for Computa- tional Linguistics.
Fuwei Zhang, Zhao Zhang, Xiang Ao, Dehong Gao, Fuzhen Zhuang, Yi Wei, Qing He. 2022. Mind the gap: Cross-lingual information retrieval with hier- archical knowledge enhancement. In Thirty-Sixth AAAI Conference on Artificial Intelligence, AAAI 2022, Sairy-Fooutth Conference on Innovative Ap- plications of Artificial Intelligence, IAAI 2022, The Twelveth Symposium on Educational Advances in Ar- tificial Intelligence, EAAI 2022 Virtual Event, Febru- ary 22 March 1, 2022, pages 4345 4353. AAAI Press.

---

A
Appendix
Language Ayoola (2022)
(2022)
mReFinED
91.0
95.8
96.6
de
89.9
93.9
95.7
A.1 Setup
86.3
95.9
96,7
We trained model 11 languages: ar, de, en, es, fa, fr, it, ja, ta, and During training, we seg- ment the training document into chucks consisting 300 tokens each, and limited the mention per chunk only 30 mentions. We use two layers of mBERT as the description encoder. We train mRe- FinED for eight days using the AdamW optimizer, with learning rate of 5e 5e-4, and a batch size of 64 for two epochs 8 A100 (40 GB). For hyper- parameter settings, we the hyper-parameters as addition, evaluate the re-
90.1
89.6
94.4
fa
90.7
90.1
92.3
ja
90.5
90.1
91.5
St
91.4
96.1
97.2
88.7
91.4
95.1
macro-avg
94.7
Table 4: Candidate generation score (recall) on Mewsli- We only top-30 candidates.
Language Ayoola al. (2022)
Cao et (2022)
mReFinED
85.3
83.2
67.0
87.0
shown in Table call score of the development set in every 2,000 steps to save the best model.
81.9
59.6
83,0
79.6
76.7
84.6
macro-ave
83.7
69.4
85.0
Table 5:
Candidate generation score (recall) on We use only top-30 candidates.
Hyper- pppem-parreete
Value
TR201 6hard,
0.01
22
X 0.01
found that increasing the number of training data more than 200k samples does not increase the per- formance of MD significantly.
T 0.02
Table 3: Hyper-parameterr of mReFinED.
61.00
Exact match score
A.2 Candidate Generation (CG) Results
60.75
In this experiment, we demonstrated the recall score of various candidate generation methods on both MEL datasets and why we need to combine two CG. We adopt ReFinED's CG from monolin- gual multilingual PEM tables using multilin- gual Wikipedia data and articles. As shown in Table 4, mGENRE's CG outperformed ReFinED's CG by 3.4 points on the average case. This is because mGENRE's CG was formulated from Wikipedia in 2019, while ReFinED's CG was for- mulated in 2022, and there are many rare candi- dates that are not found in ReFinED's CG but ap- pear in mGENRE's CG. On the other hand, Table demonstrates that ReFinED's CG outperformed mGENRE's CG on the TR2016hurd hard dataset. Thus, combining two CGs outperforms using only one CG on both MEL datasets.
60.50
60.25
0.01
0.02
0.10
0.20
0.50
1.00
#Training Data
1c6
Figure 2: Effect of training size in the bootstrapping MD framework.
A.4 Incorrect Labels TR2016 Results
This study demonstrates the performance of mGENRE and mReFinED on the cleaned 6R2016hard dataset. As shown in Table 6, we ob- 1.5 points improvements from the original result 1 (28.4). Lastly, the number of entity men- tions was decreased from 16,357 15,380.
A.3 Bootstrapping MD Results
A.5 XTREME MD Results
In this study, demonstrated the effect of train- ing size on bootstrapping MD framework. For the training data sizes, we set the size as follows: 10k, 20k, 100k, 200k, 500k, 1M. As shown in Figure 2, the training data has affected the performance of the bootstrapping MD framework. However, we
To evaluate the performance of mention detection in mReFinED, run experiment on XTREME NER dataset by converting the labels mention detection task with BIO tags. We chose subset of languages from XTREME NER dataset as our Bootstrapping MD is trained on 9 languages

---

Cleansed TR2016 hard
Language
MD+mGENRE mReFINED
de
31.0
29.1
es
33.1
36.8
fr
26.6
26.7
it
26.3
27.1
macro-avg
29.3
29.9
Table 6: Recall score the cleansed TR2016han dataset (removed entities that linked to disambiguation pages).
Mewsli-9 dataset. Language sr is dropped here be- cause it's not available in XTREME NER dataset. Results of Fl scores are shown in table 7. Please note that FI scores of Hu et al. (2020) for NER task with three entity types LOC, PER and ORG. The fine-tuned models used to produce results in their paper are not released so we couldn't pro- duce the results for mention detection task. But we want to include their NER results here to better understand our Bootstrapping MD's performance on multilingual mention detection task and we ex- pect to achieve comparable results competitive multilingual NER models in the literature if trained on NER data.
XTREME
Model
Boosstrapping MD
69.2
74.6
73.0
750
718
44.5
729
81.4
70.7
Table 7: F1 score of mention detection on XTREME NER dataset.
A.6
GenBench Evaluation Card
Motivation
Practical
Cognitive
Intrinsic
Faireess
Generalisation type
Compo- sitional
Cross
Cross
Croxs Domain
Robust-
Structural
Task
Language
ness
Shift type
Covariate
Label
Full
Assumed
D
Shift source
Naturally
Partitioned
Generated shift
Fully generated
occuring
natural
Shift locus
Train test
Finetune
Pretrain troin
Pretrain test
truin- test
0

---

Mewsli99
TR2016
Model
de
la
ta
micro
micro
de
es
fr
i
macno
MD+aGENRE Precision
11.3
18.7
15.9
17.7
12,4
20.8
12.9
5.7
14.8
14.5
16.2
1.4
1.2
1.1
1,3
1.2
MD+mGENRE Recall
59.0
67.9
62.1
67.5
54.0
38.3
83.7
34.8
46.4
57.1
63.6
30.5
31.5
23.4
25.5
27.7
MD+mGENRE
29.4
25.4
28.1
20.2
27.0
22.3
22.5
22.6
25.8
2.7
2.3
2.0
2.5
2.4
mReFinED Precision
28.4
17.0
21.0
1.6
1.7
1.5
1.7
1.6
mReFinED Recall
64.2
68.0
84.5
33.7
49.8
65.5
28.2
34.4
25.3
25.8
28.4
mReFinED FI
24.0
32.3
29.0
32.1
24.1
34.3
28.3
15.0
29.6
27.6
30.3
3.1
3.2
2.9
3.2
3.1
Table 8: Precision, Recall and F1 scores Mewsli-9 and TR2016 datasets. We report both datasets' results of entity disambiguation and entity linking tasks. We use the bootstrapping MD model as mention detection for
mGENRE. The low precision and F1 scores are due unlabelled entity mentions MEL datasets as discussed in Section 3.4
Mewsli-9
TR2016
Model
ar
cn
C5
la
ja
SE
la
tr
macro
Ir
II
macro
SpuCy
50.6
56.0
59.3
21.3
4.1
32.7
42.0
42.1
44.3
45.5
XTREME
49.8
71.7
70.6
40.6
12.6
80.3
24.4
46.4
51.1
43.3
51.0
40.9
46.9
WikiNFuRRl
52.0
72.2
68.9
75.1
37.6
56.3
42.4
46.6
45.7
Bootstrapping
60.7
72.2
69.5
73.9
55.5
43.6
87.3
36.6
49.9
61.0
53.3
58.4
48.0
51.3
52.8
mReliinDD
63.5
75.5
71.7
75.8
57.2
47.7
89.4
36.4
56.7
63.8
51.7
59.0
47.6
50.4
52.4
Table 9: Exact match score of mention detection on Mewsli-9 and TR201 datasets. We omitted unsupported languages for each model with "-".
Ground truth
mReFinED's prediction
NOAA says Earth's oceans becoming more acidic
According to study performed by the National Oceanic and Atmospheric Administration's (NOAA) Pacific Marine Environmental Laboratory, the level of acid in the world's oceans rising, decades before scientists expected the levels to rise.
The study was performed on the coastal waters of the Pacific Ocean from Baja California, Mexico to Vancouver, British Columbia, where tests showed that acid levels some areas near the edge of the Continental Shelf were high enough to corrode the shells of some sea creatures as well as some corals. Some areas showed excessive levels acid less than four miles off the northern California coastline in the United States,
"What we found ... was truly astonishing. This means ocean acidification may be seriously impacting marine life the continental shelf right now. The models suggested they wouldn't be corrosive at the surface until sometime during the second half of this century," said Richard A. Feely, an oceanographer from the NOAA.
The natural processes of the seas and oceans constantly clean the Earthi's air, absorbing 1/3 to 1/2 of the carbon dioxide generated by humans. As the oceans absorb more of the gas, the water becomes more acidic, reducing the amount of carbonate which shellfish such as clams and oysters use to form their shells, and increasing the levels of carbonic acid. Although levels are high, they are not yet high enough to threaten humans directly.
"Scientists have also seen a reduced ability of marine algae and free-floating plants and animals to produce protective carbonate shells," added Feely.
Feely noted that, according to the study, the oceans and seas have absorbed more than 525 billion tons of carbon dioxide since the Industrial Revolution began.
Figure 3: The unlabelled entity mention samples from Section 3.4


Reviews:
[{"review_id": "1pKQ4ZRmxZ", "reviewer": "Reviewer_N6KL", "summary": "", "questions": "Is there any comparison between the bootstrapped MEL model and the original NER model? At least on only the original NER entity types.\nThe term \"end-to-end\" is very confusing, as far as I understood, almost every previous work also identify mentions, either using existing NER or Wikipedia markups as labelled data. So why does it claim to be the first \"end-to-end\"?", "limitations": "", "rating": 3, "confidence": 4, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "(1) Is there any comparison between the bootstrapped MEL model and the original NER model? At least on only the original NER entity types.\n\nWe thank you for the suggestion. The result of entity mention detection model without bootstrapping is similar to the result of entity linking without bootstrapping in table 1 so we didn’t include it in table 7 in Appendix where we compare our entity mention model with bootstrapping to SpaCY, XTREME and WikiNEuRal. Our work focuses on entity linking not NER so we trained entity mention detection model which doesn’t further distinguish among various entity types, we will add a row in table 7 to show result of entity mention detection model without bootstrapping in revised version.\n\n(2) The term \"end-to-end\" is very confusing, as far as I understood, almost every previous work also identify mentions, either using existing NER or Wikipedia markups as labelled data. So why does it claim to be the first \"end-to-end\"?\n\nOur work focuses on entity linking not NER, it is true that previous MEL work referenced in our paper use Wikipedia markups as labelled data in their training, but entity mentions were assumed given at inference time along with raw text input. We don’t consider previous MEL work end-to-end because of this assumption. Our work doesn’t make this assumption and identifies both entity mention and id given just raw text input hence the claim of the first end-to-end MEL.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "Experiments showed that mREFinED outperformed previous works, namely SpaCy, XTREME, and WikiNEuRal.", "reasons_to_reject": "Is there any comparison between the bootstrapped MEL model and the original NER model? At least on only the original NER entity types.\nThe term \"end-to-end\" is very confusing, as far as I understood, almost every previous work also identify mentions, either using existing NER or Wikipedia markups as labelled data. So why does it claim to be the first \"end-to-end\"?", "questions_to_authors": "Is there any comparison between the bootstrapped MEL model and the original NER model? At least on only the original NER entity types.\nThe term \"end-to-end\" is very confusing, as far as I understood, almost every previous work also identify mentions, either using existing NER or Wikipedia markups as labelled data. So why does it claim to be the first \"end-to-end\"?", "paper_topic_and_main_contributions": "In this paper, the authors proposed a framework for end-to-end multilingual entity linking (MEL). The framework, named mREFinED, consists of a bootstrapping mention detection, an entity typing, an entity description, an entity disambiguation, and combine all in a multi-task training objective. Experiments showed that mREFinED outperformed previous works, namely SpaCy, XTREME, and WikiNEuRal.\nIs there any comparison between the bootstrapped MEL model and the original NER model? At least on only the original NER entity types.\nThe term \"end-to-end\" is very confusing, as far as I understood, almost every previous work also identify mentions, either using existing NER or Wikipedia markups as labelled data. So why does it claim to be the first \"end-to-end\"?", "excitement": 3, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "ezz5e6o7gq", "reviewer": "Reviewer_4Pgx", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "(1) The paper fails to offer a comparative analysis with a selection of mainstream large language models currently available, such as ChatGPT, GPT-4, and LLAMA. This omission hampers a well-rounded performance assessment of the proposed methods.\n\nWe thank you for the suggestion. For the LLMs experiment, we think it is not realistic to expect a LLM to store an entire KG for candidate retrieval unless backed by an API, we also agree with the opinions of Anna Rogers and other in the NLP community that evaluating against closed LLMs is not ideal for various reasons (see https://hackingsemantics.xyz/2023/closed-baselines/)\n\n (2) The predominance of zero values in the 'w/o bootstrapping' test results is troubling. The vast discrepancy between these results and the ME test results fails to convincingly demonstrate the efficacy of bootstrapping method, leading me to question the validity of these tests. I recommend that more extensive testing be carried out across a broader range of datasets to better validate the benefits of the bootstrapping approach.\n\nAs stated in Section 3.2, in the ‘w/o bootstrapping’ setting, we omitted the bootstrapped mentions in our training data to show that, without the additional mentions, the model was confused to detect mention spans since there are unlabeled mention spans in the training data (as shown in Figure 3). In addition, we ran an additional mention detection experiment on XTREME NER benchmark datasets and found that the performance of our MD and SOTA is comparable. We will include this new experiment in the camera-ready version.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "(1) This paper pioneers the introduction of the first-ever end-to-end multilingual entity linking (MEL) within a unified model, effectively expanding the reach of ReFinED into multilingual ReFinED. This represents a consequential leap forward within the discipline, providing a holistic solution for MEL. \n(2) The authors augment their contribution by suggesting a novel bootstrapping mention detection framework. This ingenious methodology rectifies the widespread challenge of unlabelled mentions in end-to-end MEL, amplifying both the precision and speed of entity linking processes. ", "reasons_to_reject": "(1) The paper fails to offer a comparative analysis with a selection of mainstream large language models currently available, such as ChatGPT, GPT-4, and LLAMA. This omission hampers a well-rounded performance assessment of the proposed methods. \n(2) The predominance of zero values in the 'w/o bootstrapping' test results is troubling. The vast discrepancy between these results and the ME test results fails to convincingly demonstrate the efficacy of bootstrapping method, leading me to question the validity of these tests. I recommend that more extensive testing be carried out across a broader range of datasets to better validate the benefits of the bootstrapping approach. ", "questions_to_authors": "", "paper_topic_and_main_contributions": "The text explores the concept of end-to-end multilingual entity linking (MEL), a procedure that pinpoints multilingual entity mentions and associates them with the relevant entity IDs within a knowledge base. Current approaches circumvent the entity mention detection stage, attributable to the scarcity of high-quality multilingual training datasets. To rectify this shortfall, the authors introduce mReFinED, the inaugural comprehensive MEL system. They further propose an innovative bootstrapping mention detection framework designed to enhance the caliber of training corpora. Empirical data illustrates that mReFinED outshines its predecessors in MEL assignments, while simultaneously offering an efficiency improvement of 44 times. ", "excitement": 3, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "1GFHubwcSK", "reviewer": "Reviewer_9qjt", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 3, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "(1) One of the main weaknesses of the paper is that neither F1 or precision are looked at in either the analysis or appendix. I understand following prior work to report only recall, but for a system to be usable it must also perform well on a precision-oriented metric. Given the lack of analysis in terms of precision, it makes me wonder how usable the mReFinED model is.\n\nWe thank you for the valuable suggestion. As stated in section 3.4, both Mewsli-9 and TR-2016 datasets suffer unlabelled entity mention problem, this will penalise precision despite correct entity mention and linking predictions are made, there is an example in figure 1 in Appendix. For the revised version, we will include F1 and precision scores. \n\n(ii) A second weakness is the choice to analyze the dataset rather than analyze the algorithm in Section 3.4. Evaluation data does not need to be perfect as long as it is not systematically biased against or in favor on any one particular system since such errors will depress the scores of all systems that utilize the dataset. It is important to remember that evaluation is about determining relative performance of systems rather than absolute values. I would much prefer such dataset analysis in the appendix, and to instead include a qualitative analysis of the algorithm.\n\nFor the design choice of our framework, we carefully choose each component to achieve SOTA based on the study of ReFinED, such as the design analysis of entity prior, entity type, and description training objectives. In the camera-ready version, we will include this experiment in Section 3.5. \n\n(3) Typos Grammar Style And Presentation Improvements\n\nIn the revised version, we will address all comments, including how we handle overlapping entities.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "A major strength of this paper is proposing the first end-to-end MEL system that generally performs better than the two-stage approach in terms of recall in most languages. The description of the approach is clear, and there is sufficient detail to reproduce the proposed method of bootstrapping strapping. ", "reasons_to_reject": "One of the main weaknesses of the paper is that neither F1 or precision are looked at in either the analysis or appendix. I understand following prior work to report only recall, but for a system to be usable it must also perform well on a precision-oriented metric. Given the lack of analysis in terms of precision, it makes me wonder how usable the mReFinED model is.\n\nA second weakness is the choice to analyze the dataset rather than analyze the algorithm in Section 3.4. Evaluation data does not need to be perfect as long as it is not systematically biased against or in favor on any one particular system since such errors will depress the scores of all systems that utilize the dataset. It is important to remember that evaluation is about determining relative performance of systems rather than absolute values. I would much prefer such dataset analysis in the appendix, and to instead include a qualitative analysis of the algorithm. \n", "questions_to_authors": "", "paper_topic_and_main_contributions": "This paper proposes the first technique for creating an end-to-end multilingual entity linker (MEL), mReFinED based on monolingual version ReFinED. Prior MELs only attempted the linking started with the from identified mentions. A key part to getting ReFinED to succeed at the task was to create a bootstrapped dataset to annotate unlabeled mentions in Wikipedia. The approach was evaluated on two datasets, Mewsli-9 and TR2016^hard. The end-to-model outperforms two-stage models on both datasets.", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Typos_Grammar_Style_and_Presentation_Improvements": "Mention how you handle overlapping entities in Section 2\n\nTable 2: Is MD missing from mGENRE?\nL347 codes -> code\n"}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper presents a study that is able to perform end-to-end multilingual entity linking. The reviewers are generally positive on this work, especially given it's a first end-to-end approach to such a problem. One concern from the reviewers is that the evaluation can be further strengthened (e.g., with more evaluation metrics). Overall this is a good short paper that contains interesting research progress that is worth sharing with the community.
Decision: Accept-Findings