Increasing Coverage and Precision of Textual Information in Multilingual Knowledge Graphs
Authors: Simone Conia
Min Li
Daniel Lee
Umar Farooq Minhas
Ihab Ilyas
Yunyao Li
Venue: EMNLP 2023 Main (2023)
Keywords: knowledge graph
multilingual
entity linking
knowledge graph completion

Abstract: Recent work in Natural Language Processing and Computer Vision has been using textual information – e.g., entity names and descriptions – available in knowledge graphs to ground neural models to high-quality structured data. However, when it comes to non-English languages, the quantity and quality of textual information are comparatively scarce. To address this issue, we introduce the novel task of automatic Knowledge Graph Completion (KGE) and perform a thorough investigation on bridging the gap in both the quantity and quality of textual information between English and non-English languages. More specifically, we: i) bring to light the problem of increasing multilingual coverage and precision of entity names and descriptions in Wikidata; ii) demonstrate that state-of-the-art methods, namely, Machine Translation (MT), Web Search (WS), and Large Language Models (LLMs), struggle with this task; iii) present M-NTA, a novel unsupervised approach that combines MT, WS, and LLMs to generate high-quality textual information; and, iv) study the impact of increasing multilingual coverage and precision of non-English textual information in Entity Linking, Knowledge Graph Completion, and Question Answering. As part of our effort towards better multilingual knowledge graphs, we also introduce WikiKGE-10, the first human-curated benchmark to evaluate KGE approaches in 10 languages across 7 language families.

Increasing Coverage and Precision of Textual Information in Multilingual Knowledge Graphs
Simone Conia"
Min Li
Daniel Lee"
Sapienza University of Rome simone. www.ioonnniiinnn.. it
Apple
University of Calgary daniel. www.lleeeeelalaaa @ucalgary.ca
jii__i6@apples com com
Umar Farooq Minhas Apple ufminhas@@pple..cm
Ihab llyas
Yunyao Li
Apple ii iilyas@apple.com com
Apple yunyaoli@apple.com com
Abstract
is a semantic relationship that represents a fact (e.g., "Rome is the capital of Italy" or "The Leg- end of Zelda is video game series"). With the wealth of information that knowledge graphs pro- vide, they play a fundamental role in multitude of real-world scenarios, touching many areas of Artificial Intelligence (Nickel et al., 2016), includ- ing Natural Language Processing (Schneider et al., 2022), Computer Vision (Marino et al., 2017), In- formation Retrieval (Reinanda et al., 2020), and recommender systems (Guo et al., 2022).
Recent work in Natural Language Processing and Computer Vision has been using textual information e.g., entity names and descrip- tions available in knowledge graphs to ground neural models to high-quality structured data. However, when comes non-English lan- guages, the quantity and quality of textual in- formation are comparatively scarce. To address this issue, we introduce the novel task of auto- matic Knowledge Graph Enhancement (KGE) and perform thorough investigation on bridg- ing the gap in both the quantity and quality of textual information between English and non- English languages. More specifically, we: bring to light the problem of increasing multi- lingual coverage and precision of entity names and descriptions Wikidata; ii) demonstrate that fatt--of-the-art methods, namely, Machine Translation (MT), Web Search (WS), and Large Language Models (LLMs), struggle with this task; iii) present M-NTA, novel unsupervised approach that combines MT, WS, and LLMs generate high-quality textual information: and, iv) study the impact of increasing multi- lingual coverage and precision non-English textual information in Entity Linking, Knowl- edge Graph Completion, and Question Answer- ing. As part of our effort towards better mul- tilingual knowledge graphs, also introduce WikiKGE-10, 10, the first human-curated bench- mark to evaluate KGE approaches in 10 lan- guages across 7 language families.
Over the years, knowledge graphs have mainly been adopted as a rich source of human- re- lational information to enhance neural-based mod- els for tasks of varying nature (Huang et al., 2019; Bevilacqua and Navigli, 2020; Orr et al., 2021). However, ever since natural language text has proven to be an effective interface between struc- tured knowledge and language models (Guu et al., 2020; Petroni et al.. 2019; Peng et al., 2023a), the value of knowledge graphs has become twofold: besides providing relational information, knowl- edge graphs have also become reliable source of high-quality textual information. Indeed, recent approaches have been increasingly reliant on tex- tual information from knowledge graphs to surpass the state of the art (Barba et al., 2021; Chakrabarti et al., 2022; De Cao et al., 2022; Xu et al., 2023).
Unfortunately, when comes to -nnglishsh languages, the condition of multilingual textual in- formation in knowledge graphs is far from ideal. Indeed, popular resources present a significant gap between English and non-English textual informa- tion, hindering the capability of recent approaches to scale to multilingual settings (Peng et al., 2023b) Importantly, this gap exists in high-resource lan- guages even if we consider basic textual properties, such as entity names and entity descriptions. The nature of the problem is dual: disparity in coverage, as the quantity of textual information available in
1 Introduction
The objective of a knowledge graph is to encode our collective understanding of the world in well- defined, structured, machine-readable representa- tion (Hogan et al., 2021). At high level, each node of knowledge graph usually represents a concept (c.g., universe, weather, or president) or an entity (c.g., Albert Einstein, Rome, or The Leg- end of Zelda), and each edge between two nodes
Work done as intern at Apple.

---

non-Einglish languages is more limited, and preci- sion, the quality of non-English textual informa- tion is usually lower.
Knowledge graphs. Livnn though their exact def- inition remains contentious, knowledge graphs are usually defined as "a graph of data intended to accumulate and convey knowledge of the real world, whose nodes vepresent entities of interest and whose edges represent potentially different re- lations between these entities" (Hogan et al., 2021), Over the years, research endeavors in knowledge graphs have steadily focused their efforts primar- ily on using their relational information, i.e., the semantic relations between entities. Besides foun- dational work on knowledge graph embedding tech- niques, which represent the semantics of an entity by encoding its graph neighborhood (Wang et al., 7), relational knowledge has been successfully employed in Question Answering to encode proper- ties that generalize over unseen entities (Bao et al., 2016; Zhang et al., 2018: Huang et al., 2019), in Text Summarization to identify the most relevant entities text and their relations (Huang et al., 2020; and Zhao, 2021), in Entity Linking to condition the prediction of an instance on knowl- edge subgraphs (Raiman and Raiman, 2018; Orr et al., 2021), and in Word Sense Disambiguation to produce rich meaning representations that can differentiate closely related senses (Bevilacqua and Navigli, 2020: Conia and Navigli. 2021).
In this paper, we address the aforementioned cov- erage and precision issues of textual information in multilingual knowledge graphs via a data-centric approach. Our contributions include the following:
We introduce the task of automatic Knowl- edge Graph Enhancement (KGE) to tackle the disparity of textual information between English and non-English languages in multi- lingual knowledge graphs:
We present WikiKGE-10, a novel human- curated benchmark for evaluating KGE sys- tems for entity names in 10 typologically di- verse languages: English, German, Spanish, French, Italian, Simplified Chinese, Japanese, Arabic, Russian, and Korean;
We investigate how well Machine Translation (MT), Web Search (WS), and Large Language Models (LLMs) narrow the gap between English and non- Enngish languages.
We propose M-NTA, novel unsupervised ap- proach, which combines MT, WS, and LLMs to mitigate the problems that arise when using each system separately;
Textual information in knowledge graphs. While knowledge graphs have been used for the versatility of their relational information, the rapid emergence of modern language models has also represented turning point in how the research community looks at knowledge graphs. As mat- ter of fact, the initial wave of Transformer-hase based language models (Devlin al., 2019; Radford al., 2019 9) were trained purely on text, and, when re- searchers realized that quantity and quality of train- ing data are two essential factors to enable better generalization capabilities (Liu et al., 2019), it be- came clear that the textual data available in knowl- edge graphs could be exploited as direct interface between human-curated structured information and language models.
We demonstrate the beneficial impact of KGE in downstream tasks, including Entity Link- ing, Knowledge Graph Completion, and Ques- tion Answering.
We deem that achieving parity coverage and precision of textual information across languages in knowledge graphs is fundamental to enable better and more inclusive multilingual applications. In the hope that contributions can set stepping stone for future research in this field, we release WikiKGE- 10 at
2 Related Work
In this section, we provide a brief overview of knowledge graphs, highlighting how textual infor- mation from knowledge graphs is now as important their relational information, showcasing how recent work has successfully integrated textual in- formation into downstream applications, and re- viewing how recent efforts have mainly focused on completing relational information in knowledge graphs rather than textual information.
Indeed, prominent knowledge graphs Wiki- data (Vrandečić and Krötzsch, 2014), DBPe- dia (Lehmann al., 2015), YAGO (Hoffart et al., 201 1), and BabelNet (Navigli et al., 2021), among others feature lexicalizations for each entity in multiple languages, e.g., names, aliases and de- scriptions of various length. Therefore, textual in- formation in knowledge graph is now as important

---

relational information, with recent developments taking advantage of the former to surpass the previ- ous state of the art in an increasingly wide array of tasks, such as Word Sense Disambiguation (Barba et al., 2021), Entity Linking (Xu et al., 2023; Proco- pio et al., 2023), Relation Alignment (Chakrabarti et al., 2022), and Language Modeling itself (Xiong et al., 2020; Agarwal et al., 2021: Li et al., 2022a; Liu et al., 2022). Unfortunately, the wide adoption of such techniques in multilingual settings has been strongly limited by the disparity in coverage and quality of entity names and descriptions in mul- tilingual knowledge graphs between English and non-English languages (Peng et al.. 2023b).
in updating textual information across languages.
3.1 Task definition
Given an entity e in knowledge graph G, we de- fine Knowledge Graph Enhancement (KGE) as the task of automatically producing textual information about e for each language E L, where L is the set of languages of interest. More precisely, KGE encompasses two subtasks:
Increasing coverage of textual information, which consists in providing textual informa- tion that is currently unavailable for in G;
Increasing precision of textual information, which consists in identifying inaccurate under-specified facts in the textual informa- tion already available for e in G.
Knowledge graph acquisition and completion. Finally, we would like stress that our endeavor is orthogonal to the efforts that usually fall under the umbrella terms of "knowledge acquisition" (Ji et al., 2022) and "knowledge graph completion" in the literature (Lin et al., 2015; Shi and Weninger, 2018; Chen et al., 2020b). More specifically, the ob- jective of these two tasks is to construct the "struc- ture" of knowledge graph, i.e., identifying the set of entities of interest and the (missing) relations between entities. Therefore, the multilingual ex- tensions of these two tasks are concerned about detecting missing nodes or edges in a multilingual knowledge graph (Chen et al., 2020a; Huang et al., 2022; Chakrabarti et al.. 2022), whereas we specif- ically focus on expanding the coverage and preci- sion of textual information in multilingual knowl- edge graphs. Nonetheless, we argue that increas- ing coverage and quality of textual information in multilingual knowledge graphs has beneficial cascading effects on tasks like knowledge graph completion, as our experiments show Section
Therefore, KGE evaluates the capability of sys- tem to provide new textual information (coverage) as well as its capability to detect errors and inac- curacies in existing textual information (precision). While textual information may refer to any entity property expressed in natural language, in the re- minder of this paper, we focus entity names and entity descriptions in Wikidata, which have become increasingly used in knowledge-infusee language models and state-of-the-ard systems (see Section 2).
3.2 Coverage of non-English information
Ideally, we would like every entity e in Wikidata to be "covered" in all languages, i.e., we would like Wikidata to provide name and description of for each / in the set I, of the languages sup- ported by the knowledge graph. In practice, this is not the case in Wikidata, as we can observe in Figure which provides bird's-eye view on the availability of entity names and entity descriptions in 9 non-English languages. More precisely, we analyzed the Wikidata entities that have an associ- ated Wikipedia page' with at least 100 page views in any language over the 12 months between May 2022 and April 2023. Our analysis calls attention to the issue of coverage of entity names and en- tity descriptions in Wikidata, which is significant even if we only consider head entities - top- of the most popular entities sorted by number of Wikipedia page views - and restrict the set of lan- guages to German, Spanish, and French, which are
3 Knowledge Graph Enhancement of Textual Information
While relational information in knowledge graphs is usually language-agnostic (e.g., "AI" is a field of "Computer Science" independently of the lan- guage we consider), textual information is usu- ally Lunguage-dependees (e.g., the lexicalizations of "Al" and "Computer Science" vary across lan- guages). With the growing number of languages supported by knowledge graphs, it is increasingly challenging for human editors maintain their content up-to-date in all languages: therefore, we believe it is important to invest in the development and evaluation of systems that can support humans
'The Wikidata-to- Wikipedia mapping is n-to-1 since a Wikidata entity may refer to the entire Wikipedia article or a section of an article.

---

Head entities
Torso entities
Tall entities
100%
27
24
Names
Coverage
58
73
79
88
84
63
80
40
64
36
49
45
85
39
61
54
37
80
42
72
36
87
80
86
RP
0%
AR DE
ES
FR
KO RU ZH
ES FR
JA
JA KO RU ZH
100%
Descriptions
Coverage
48
86
34
65
36
66
50
80
48
59
41
59
18
58
44
6B
60
76
84
84
B3
19
16
18
17
25
20
0%
AR DE ES KO RU ZH
ES FR IT
JA KO RU
Coverage (%)
Headroom EN (%)
Figure 1: Coverage of mon-Engllish textual information entity names and descriptions compared to English in Wikidata. Even for head entities (top- 10% in terms of Wikipedia page views), there is large disparity between English and non-Emn En---Enlishh coverage; the situation is unexpectedly worse on torso (top-50%) and tail entities, Best seen in color.
usually regarded "high-resource" languages. Un- surprisingly, we can observe that the gap cover- age increases when we consider entities belonging to the torso (top-50%) and tail of the popularity dis- tribution, as the coverage of Japanese and Chinese names for tail entities is lower than 15%.
inaccurate information into three main classes:
Human mistakes, when the imprecision was caused by human editor. For example, entity Q1911 is incorrectly named Oliver Giroud in Spanish instead of Olivier Giroud.
We argue that the fact that Wikidata inherits this disparity from Wikipedia, which is edited by disproportionate number of English-speaking con- tributors. should not detract our attention from this issuc. As a matter of fact, growing num- ber of approaches relies on textual information from Wikidata; therefore, we believe that the stark contrast between today's great interest for textual information in knowledge graphs and the scarce multilingual coverage revealed by our analysis mo- tivates the development of "data-centric Ar" ap- proaches (Zha et al., 2023) for increasing multi- lingual coverage, rather than focusing our efforts exclusively on model-centric novelties.
Stale entries, when new information is avail- able but Wikidata has not been updated. For example, the English description of entity Q927916 6 has been recently updated include the date of death but the Russian description still indicates the date of birth only.
Under-specifie information, when the avail- able information is not incorrect but it is still too generic. For example, the Spanish description for Q345494 is "másico japonés" (Japanese musician), whereas the German is "japanischer Komponist, Pianist, Produzent und Schauspieler (195222023)" which details his work (composer, pianist, producer, and ac- tor) and includes his birth and death dates.
3.3 Precision of non-English information
While non-Enngish coverage of entity names and entity descriptions is critical, another crucial as- pect is the level of precision in Wikidata. Indeed, the majority of the approaches that rely names and descriptions often use such information as-is and overlook the possibility that it may be inaccu- rate. More specifically, we categorize the causes of
Although it is not uncommon to encounter in- stances of these three classes of error in Wikidata, conducting a comprehensive analysis of its entire knowledge graph is unfeasible.
3.4 Evaluating KGE with WikiKGE-10
To address the above-mentioned issues, we present WikiKGE-10, novel resource for benchmark- ing data-centric-Al approaches on KGE of entity
"The primary language of Wikipedia editors is English (52%), followed by German (18%), Russian and Spanish (both at 10%) [souree: UNU-Mcrit).

---

AR
DE
EN
ES
FR
IT
JA
KO
RU
ZH
All
Entitics
1,000
1,000
1,000
1,000
1,000
1,000
1,000
1,000
1,000
1,000
10,000
Entity names in WikiKGE-10
4,213
3,498
2,837
4,320
3,548
3,156
2,999
3,874
3,901
4.088
36,434
Entity in Wikidata
2,521
2,336
2,090
2,732
2,330
1,840
2,235
2,136
2,706
2,569
23,495
Entity name errors in Wikidata
320
491
219
571
530
236
486
329
507
830
4,663
Table I: Overview of WikiKGE-10, which features 10 languages Arabic (AR), German (DE), English (EN),
Spanish (ES), French (FR), Italian (IT), Japanese (JA), Russian (RU), simplified Chinese (ZH).
names in 10 languages: English, German, Span- ish, French, Italian, Chinese, Japanese, Korean, Arabic, and Russian. At a high level, WikiKGE- 10 is designed to feature typologically-diffeeent linguistic families, from West Germanic to Ro- mance, Semitic, Slavic, Koreanic, Japonie, and Sino-Tibetan, and, therefore, to enable compar- ison of entity names across a set L of 10 di- verse languages with heterogeneous, possibly non- overlapping vocabularies and scripts.
ing the coverage and precision gap between English and non-English languages. Therefore, we also introduce M-NTA (Multi-source Naturalization, Translation, and Alignment), simple unsuper- vised ensembling technique, which overcomes the limitations of MT, WS, and L.MMs by combining and ranking their predictions. Here, we direct our attention toward entity names, but we also show that the methodologies discussed in this section can be extended to other types of textual information, such as entity descriptions, in Appendix C.
Given language E L, we uniformly sam- pled 1000 entities from the top- of the entities in Wikidata sorted by the number of page views for their corresponding Wikipedia article in I. We note that the composition of the top-10% entities and, therefore, our sample of 1000 entities may significantly vary from language to language, as the popularity distribution changes according to what different cultures care about (Hershcovich et al., 2022). After selecting 1000 entities for each language, human graders manually checked their existing names to assess their correctness, while also adding new valid names. The annotation pro- cess, which took more than 2,500 human hours, resulted in around 36,000 manually-curated names across 10 languages; provide more details on the creation of WikiKGE-10 and our guidelines in Appendix A. Importantly, as shown in Table we find that human graders deemed 20% of the entity names in Wikidata to be incorrect and that 40% of the valid entity names could not be found in Wikipedia. In practice, since WikiKGE-10 features manually-curated entity names and indicates which names in Wikidata are incorrect inaccurate, our benchmark can be used to evaluate the capability of system to tackle both subtasks in KGE, i.c.. increasing coverage and precision of entity names.
4.1 Baseline approaches
Machine Translation (MT). When in need of converting information from one language to an- other, employing MT is a typical choice. Indeed, given a source language 1, and a target language 4, a straightforward approach would be to use an MT system to translate the textual information avail- able in la to le to increase coverage in 4. However, such approach is limited in several respects: it assumes that all textual information is avail- able in 4, which, in practice, is not the case even when 1, is English, i.c., MT cannot be applied if the information to translate is not available in the source language in the first place; ii) it assumes that MT systems are precise, which, again, is not the case: for example, entity names can be com- plex and ambiguous to translate without additional context (e.g., "Apple" could refer the fruit the tech company); and, iii) while MT can be em- ployed to increase coverage, is not clear how to apply MT to identify inaccurate entity names to increase precision of existing textual information.
Web Search (WS). A common workflow for looking up textual information in target language l is to query Web search engines with queries in source language l,, such as "[entity-name] in [l,"" and extract the answer from the search re- sults, possibly limiting the search space to Web pages entirely in 4 or originating from countries in which 4 is the primary/official language. While
4 Methodology
In this section, we consider three broad families of approaches- - MT, WS, and LLMs - and demon- strate their unsatisfactory performance on narrow-

---

WS can provide more varied results that are not 1-to-1 translations of the source entity name, we argue that WS suffers from the same fundamen- tal limitations as MT: i) if le is not complete, then we cannot formulate every search query; ii) WS is prone to biases, especially for ambiguous instances (e.g.. googling "plane" shows many results about airplanes, a few results about geometric planes, and none about plane trees); and, iii) using WS to iden- tify and correct imprecise textual information in a knowledge graph is not obvious.
e in l, from Wikidata and uses it to produce a natural language representation r, rs(e" ed)) of e in 4.- This allows M-NTA to rely on dif- ferent representations for polysemous words, e.g. "Apple is an American technology com- pany" and "Apple is a fruit of the apple tree."
2. Translation: next, M-NTA "translates" the representation (c e from le to using a system f()) to obtain a natural language output re(e], eg) in the target language.
Large Language Models (LLMs). Recent LLMs have been shown to be few-shot learners. thanks what is now known in-context learning, or the capability of capturing latent relationships between few input examples to provide an answer for a new task (Brown et al., 2020). With the advent multilingual LLMs, such as BLOOM (Scao et al., 2023), mT5 (Xue et al., 2021), and their instruction- fine-tuned variants (Muennighoff et al., 2022), we can prompt such models for translation, e.g.. "How do you say [entity-name] in f4/?", possibly provid- ing a few examples input to condition the gen- eration of the output. While prompting language models versatile, relying on LLMs also exposes us to their weaknesses, e.g., hallucinations (Ji et al., 2023) and data biases (Navigli et al., 2023).
3. Alignment: finally, M-NTA aligns the output re(eff ef) with the input rs(ee, e²) to extract the entity name c..
Most crucially, M-NTA is transparent to the defini- tion of a source system f()). This allows M-NTA to take advantage of any source system f() that is able to produce e/" More specifically, M-NTA can use a set of source systems = fe.-.., in which fi can be an MT, WS or LLM- based sys- tem. Not only that, we can leverage the same MT system multiple times by setting the source lan- guage to different languages, allowing M-NTA to draw knowledge from all the languages of inter- est to produce better results in 44.
The second ques- tion how to validate each view by using the other views. In practice, we first consider each view as an answer = f(-) provided by a source system f() in the set of source systems F. Then, we assign an agreement score a (yy) to each answer:
Ranking answers in M-NTA.
4.2 M-NTA: Multi-source Naturalization, Translation, and Alignment
To address the issues above, introduce M-NTA, simple unsupervised technique that combines MT, WS, and LLMs. The intuition behind M-NTA is that obtaining a fact from multiple source systems may offer complementary pieces of information which provide varying views on our world knowl- edge; we hypothesize that, if distinct views support the same fact, there is greater chance for the fact to be closer the ground truth.
(vy = E EVv.V Vy = WV=OSEFFU
where (yy. - {0, 1) a function that indicates if y supported by v/, e.g. in the case of entity names Φ(.,-) can be implemented as exact string match. In other words, the agreement score oty) is higher when an answer from a source system f is supported by an answer y' from another source system f': if is valid according to multiple source systems, then there is lower chance for y be incorrect. On the contrary, if y is not supported by other answers, its agreement score lower and, therefore, there is a higher chance for y to be incor- rect. Finally, we obtain the final set of answers Y by selecting all the answers y whose score (1) is greater than or equal to a threshold A:
Source systems in M-NTA. The first question, therefore, is how to produce the above-mentioned views on our world knowledge. Given a source language I, and an entity whose name in 1 is ea M-NTA takes three-steps approach to generate et in target language I;:
1. Naturalization: mentioned above, entity names are not suitable for direct translation since they might not provide sufficient con- text (Li et al., 2022b). To overcome this issue, M-NTA retrieves the textual description ed of
y = (v: 2 A}

---

AR
DE
EN
ES
FR
IT
KO
ZII
Aeg
C
C
C
C
C
C
C
C
C
P
P
28.1
47.1
48.3
51.6
24.1
52.1
30.8
28.3
EN
30.2
45.1
52.1
67.1
50.9
63.1
50.2
62.8
54.1
65.2
29.9
55.3
32.8
49.2
58.1
57.3
30.6
57.1
41,01
58.0
MT From
E5
27.3
43.1
16.0
63.1
37.1
58.5
18.9
60.8
52.9
61.0
27.7
56.1
32.0
47.4
36.2
55.2
27.1
53.2
37.5
55.5
27.0
37.6
48.3
58.9
52.9
64.0
27.4
54.5
52.3
35.9
55.1
23.4
53.6
37.4
55.5
IT
26.8
43.6
48.2
62.9
36.4
56.7
46.8
57.8
49.2
61.3
28.0
54.6
31.6
48.2
35.9
55.4
26.3
53.5
36.6
55.1
23.3
31.1
43.3
52.1
18.9
28.0
32.0
2H
22.7
36.2
42.0
35.7
302
49.2
39.0
48.1
40.3
51.8
44.9
58.0
19.3
44.1
26.0
39.4
29.4
48.5
32.6
47.9
M-N7A 12Mc M5
Googlesant
14.5
28.0
36.4
54.1
19.3
$2.0
39.0
57.6
43.6
53.5
16.1
44.3
25.6
58.5
29.1
47.2
18.5
36.2
28.9
45.7
mTDauge
15.2
40.3
39,4
44.2
16.5
22.4
37.0
29.4
mT0g
15.8
31.1
42.1
$1.4
41.5
54.2
39.9
58.0
44.5
$1.9
16.9
16.1
25.2
39.5
50.1
48.4
19.2
57.8
30.4
47.2
millaal
17.1
43.8
41.9
45.0
18.0
22.4
31.0
48.7
GPT:3
18:2
34.1
67.
61.9
45.3
60.2
45.4
62.2
49.4
62.2
21.4
49.1
26.0
42.7
32.3
53.5
22.1
50.8
34.2
53.3
GPT-335
27.4
42.1
50.5
66.
50.6
63.2
50.5
63.3
53.7
64.9
28.9
54.4
51.9
47.3
96.8
56.3
29.2
55.7
39.9
57.0
29.9
51.3
50.7
51.4
54.7
33.7
56.3
34.6
31.3
56.5
42.0
58.1
M-NTA oPE7
41.3
73.6
57.5
77.3
41.3
61.8
55.4
11.7
57.1
69.9
61.3
75.1
34.0
65.8
50.0
76.6
44.1
66.5
34.7
70.0
53.0
79.4
M-NTA
42.7
74,4
87.5
77.6
41.3
64.8
55.6
57.3
61.7
75.2
35.2
67.0
50.6
76.7
44.8
66.9
36.1
71.4
53.6
79,9
M-NTA 6P14
43.2
74.4
57.1
77.5
41.3
64.8
55.8
75,0
57.4
70.3
61.7
75.5
35.2
67.9
51.2
76.8
45.3
67.1
36.8
72.0
53.9
80.1
Table 2: Fl scores entity names coverage (C) and precision (P) in WikiKGE-10 for MT with NLLB-200, WS with Google Search, LLM prompting with mTO and GPT, and M-NTA. The symbol "_" is used to indicate that source and target languages are the same. Best results in bold.
where À is a hyperparameter that can be tuned to balance precision and recall of the system, with our experiments indicating that À = 2 is the most balanced choice for coverage, discussed in Ap- pendix C.
ular albeit closed LLMs, which has been proven to show strong multilingual capabilities. Finally, we evaluate M-NTA when scoring and ensembling the outputs from NLLB-200," Google Web Search, and GPT-3/3.5/4.
Differently from MT, WS, and LLMs, since each answer in Y scored and ranked by M-NTA, the application of M-NTA to KGE is straightforward. To increase coverage, we can consider Y as the result, as À > 1 allows M-NTA to remove unlikely answers; to increase precision, we can consider every value ý in the KG that is not in Y an incorrect value.
For each baseline, the input data is the set of entity names that currently exist in Wikidata in a source language (,, i.c., the entity names in l are "translated" into the target language I using MT, WS, LLM prompting or M-NTA. We note that, if Wikidata does not include at least one name for an entity in la then none of the systems mentioned above is able produce name in 4. M-NTA is able mitigate this issue by drawing information from multiple source languages the same time.
5 Experiments on KGE
Given a set of human-curated correct names from WikiKGE- 10 and a set of predicted names Y generated by a system, we compute coverage between Y and Y as following:
In this section, we evaluate our strong baselines and M-NTA on the task of KGE for entity names and discuss the results obtained on WikiKGE-10.
5.1 Experimental setup
(y(y)
Recently, there has been a surge of interest for mul- tilingual MT systems, i.c., systems that use a uni- fied model for multiple language pairs. Therefore, for the implementation of the MT baseline, we use NLLB-200 (Costa-jussà et al., 2022), state-of-the. art multilingual MT system that supports over 200 languages. For WS, it is often regarded as one of the best WS engines. For LLM prompting, we consider two popular mod- els: i) mTO (Muennighoff et al., 2022), an openly available instruction diitucttoo-fiettueedd multilingual LLM based on mT5, and ii) 3PT,3 one of the most pop-
PPVc
Y
NEY
(
TPRc =
YY]
Coverage =2 PPV TPRC
PPVc+TPRe
Google Web Search, as
where PPVc the positive predictive value, TPRc is the true positive rate, and 1x(x) is the indicator function, which returns I if x E X else 0. We compute precision in similar way, using the
GPT-4 were carried out in September 2023.
'Fxperiments with GPT-3 and GPT-3.5 were carried out hetween March and May 2023. Additional experiments with
'For each target language 4s, we M-NTA uses the transla- tions from every source language I, $ lc-

---

set of human-curated invalid names -Y and the set of names -Y predicted be incorrect by a system. Note that, to enable a direct and fair comparison, we allow every system to rely on additional contex- tual information in the form of entity descriptions from Wikidata; we provide more details about the experimental setting in Appendix C.
similarity is often a robust strategy for information retrieval, but, in this case, introduces significant noise, which is undesirable knowledge graph.
Prompting LLMs requires caution. Our experi- ments also indicate that prompting LLMs a better option than WS in terms of performance, especially when using GPT. However, we shall keep in mind not to take benchmark results at face value (Maru et al., 2022): analyzing the answers shows one is- sue that does not surface in numerical results is that some errors in the predictions provided by LLMs can be significantly worse and, therefore. potentially more problematic than those made by MT' and WS systems. We observe that, especially for uncommon entities and smaller models, L.Ms may produce answers that are completely unrelated to the correct answer, including copying part of the prompt or its examples, providing entity names for entirely different entities (e.g., Silvio Berlusconi (Italian politician) for San Cesario sul Panaro (Ital- ian comune)), hallucinating facts (e.g., adding that The Mandalorian (2nd season) is from Star Wars: La venganza de los Sith in Spanish), and also gen- erating nonsense outputs. It follows that, although LLMs are generally better than WS, the risk of using them is higher in case of error, purely nu- merical metrics, such coverage and precision. may hide that some errors worse than others, i... potentially more harmful in downstream appli- cations.
5.2 Results and discussion
The results on WikiKGE-10 reported in Table 2 highlight two key findings: i) our proposed solu- tion, M-NTA, offers superior performance com- pared to state-of-the-art techniques in MТ, WS, and LLMs on both coverage and precision of entity names; and, ii) the results on WikiKGE- 10 indicate that KGE is a very challenging task and that more extensive investigations are needed to design better KGE systems. In the following, we report the main takeaways from our experiments.
Different languages hold different knowledge. Our experimental results show that generating en- tity names in non-English languages by translating English-only textual information does not provide the best results, as shown in Table This is true not only for the MT system we use in our exper- iments but also for WS and LLMs, for which we use English-only queries and prompts, respectively. In particular, it is interesting to notice that com- pletely different systems, namely, MT and GPT- 3.5, produce similar results on average: 41.0% vs. 39.9% in coverage and 58.0% vs. 57.0% in preci- sion. Therefore, we hypothesize that the significant gain in performance by M-NTA - +12% cov- erage and +22% in precision over GPT-3.5 - is mainly attributable to its effectiveness in combin- ing information across different languages. Indeed, it is interesting to notice that using GPT-4 instead of GPT-3.5 one of the sources of M-NTA only provides marginal improvements to the overall re- sults in both coverage and precision.
6 Enhancing Textual Information in KGs: Impact on Downstream Tasks
In this section, we demonstrate the beneficial im- pact of KGE on downstream tasks and its effec- tiveness in improving the performance of state-of- the-art techniques in multilingual Entity Linking and Knowledge Graph Complction; we also show that KGE is beneficial for multilingual Question Answering in Appendix E.
WS may not be suitable for KGE. The results from our experiments show that WS is the least effective approach to generate entity names. Al- though we are not disclosed on the inner workings of proprietary search engines, we can qualitatively observe that the results returned from Web searches often include answers for entities that are seman- tically similar the one mentioned in the input query. For example, searching Niki Lauda (former Fl driver) in Italian also returns results about Rush (biographical film on Lauda). Relying on semantic
Multilingual Entity Linking (MEL). A direct application of increasing the quantity and qual- ity of textual information in knowledge graph is MEL., the task of linking a textual mention to an entity in a multilingual knowledge base (Botha et al., 2020). We evaluate the impact of our work on mGENRE (De Cao al., 2022), state-of-the- art MEL system that fine-tunes mBART (Lewis et al., 2020) to autoregressively generate a Wiki- data entity name for a mention in context. As noted

---

MEL
mGENRE
mGENRE + M-NTA
A
MKGC
A-KGC
L-KGC+M-NTA
A
FR
73.4
74.1*
+0.7
EN
47.4
47.5
+0.1
IT
56.8
58.2*
+1.4
ES
64.6
66.3*
+1.7
RU
65.8
66.2*
+0,4
FR
64.4
66.0"
+1.6
ZH
52.8
55.0
+2.2
JA 62.8
64.2*
+1,4
Avg
62.2
63.3*
+1.2
Avg
59.8
61.1*
+1.3
Table 3: Comparison between mGENRE and mGENRE
Table 4: Comparison between our re-implementation of Align-KGC and Align-KGC M-NTA in terms Mean Reciprocal Rank (MRR) in multilingual Knowledge Graph Completion on DBP-5L. Best results in bold. * statistically different with p < 0.05.
M-NTA in terms of F1 score in multilingual Entity Linking on Wikinews-7. Best results in bold. x statis- tically different with < 0.05.
by De Cao et al. (2022), mGENRE generates tity names by also copying relevant portions of the input mention; however, copying not possible when the mention of the entity language for which Wikidata does not feature any names. By increasing the coverage and precision of textual in- formation in Wikidata, M-NTA provides mGENRE with broader coverage of entity names in non- English languages, aiding mGENRE's capability rely on copying mechanisms. Indeed, as see in Table 3, augmenting mGENRE with M-NTA brings an improvement of 1.2 points in Fl score on average in Wikinews-7, setting a new state-of-the- art on this benchmark.
other hand, it empirically validates our assumption that increasing coverage and precision of textual information in multilingual knowledge graphs is an effective data-centric way to unlock latent perfor- mance in current systems.
Conclusion and Future Work
In this paper, we introduced the novel task automatic Knowledge Graph Enhancement, with the objective of fostering the development and evaluation of data-centric approaches for narrow- ing the gap in coverage and precision of tex- tual information between English and non-English languages. Thanks to WikiKGE-10, our novel manually-curated benchmark for evaluating KGE of entity names in 10 languages, brought to light the unsatisfactory capabilities of machine transla- tion, web search, and large language models to bridge this multilingual gap. To this end, we intro- duced M-NTA, a novel approach to combine the complementary knowledge produced by the above techniques obtain higher-quality textual infor- mation for mon-Esglish languages. Not only did M-NTA achieve promising results on WikiKGE-I 10 but our experiments also demonstrated its benefi- cial effect across several state-of-the-ard systems for downstream applications, namely, multilingual entity linking, multilingual knowledge graph com- pletion, and multilingual question answering.
Multilingual Knowledge Graph Completion (MKGC). Another direct application of KGE is MKGC, the task of predicting missing links be- tween two entities in multilingual knowledge base (Chen al., 2020a). Similarly to MEI,, we evaluate the downstream impact our work on a re-implementationn of Align-KGC (SoftAsym), a state-of-the-art MKGC system originally proposed by Chakrabarti et al. (2022), which we rebuilt to use our entity names and descriptions to create mBERT-based entity embeddings. As shown in Table using M-NTA provide more and better entity names and descriptions allows the MKGC system to obtain a consistent improvement across non-English languages on DBP-5L (Chen et al., 2020a), i.e., +1.5 points in terms of Mean Recipro- cal Rank (MRR), excluding English. We hypothe- size that the larger part of this improvement comes from the fact that the entity descriptions generated by M-NTA are more informative, as suggested by the examples shown in Appendix C.7 (see Table 7). On one hand, this improvement demonstrates the flexibility of M-NTA, as DBP-5L is based on a different knowledge graph, i.e., DBPedia, On the
We hope that our novel benchmark and method can represent a milestone for KGE. However, work demonstrates that, if aspire to achieve quantity and quality parity across languages, we still need more extensive investigations on how effectively increase coverage and precision of tex- tual information in multilingual knowledge graphs.

---

Limitations
extent of the problem not only on general knowl- edge graphs but also on domain-specifte ones.
Textual information in knowledge graphs. In this paper, we focus on two specific types of tex- tual information, namely, entity names and entity descriptions. Although our discussion on cover- age and precision of textual information (or lack thereof) can be extended to other types of textual in- formation, e.g., longer descriptions like Wikipedia abstracts or coreferential information like the an- chor text of the hyperlinks in a Wikipedia article, our analysis in Sections 3.2 ("Coverage of non- English information") and 3.3 ("Precision of non- English information") highlights that the gap be- tween English and non-English names and descrip- tions is very large even for popular entities, ranging from 20% to 60% for entity names and from 30% to 80% for entity descriptions. Furthermore, entity names and entity descriptions are the most widely used types of textual information from knowledge graphs in downstream tasks (see Section 2), and, therefore, we decided to focus our discussion on these two types, which potentially have a more di- rect impact on downstream applications, as also shown in Section We hypothesize that most of our observations generalize to other types of tex- tual information in knowledge graphs; however, we leave deeper investigations and the creation of benchmarks for other types of textual information in knowledge graphs to future work.
WikiKGE-10. Although WikiKGE-10 covers a wide range of entities a total of 36,434 manually- curated entity names it still focuses only on enti- ties belonging to the head of the popularity distri- bution of Wikipedia. Our attention is directed to popular entities as we observed large gap of cov- erage between English and non non- -nnglishh languages even for entities that are in the top- 10%: our bench- mark shows that current state-of-the-ar techniques, namely, MT, WS, and LLMs, still struggle to pro- vide correct entity names for popular entities. We hypothesize that such techniques will also strug- gle on less popular entities, i.c., entities belonging to the torso and tail of the popularity distribution. However, we cannot assume that the performance and more importantly the ranking between MT, WS, and LLMs is the same on torso and tail en- tities, e.g., WS may be more robust than LLMs in generating names for tail entities. Future work may take advantage of the methodology presented in this paper to create benchmarks for more chal- lenging settings. Last but not least, we stress the fact that the popularity of an entity is variable over time; therefore, entities that are now in the top- t0p-10% may not be as popular in the next year, vice- versa, previously unknown entities may become extremely popular in the short-term future.
Different knowledge graphs. Our attention is mainly directed at Wikidata, as it one of the most popular multilingual knowledge graphs used by the research community in Natural Language Pro- cessing as well as Information Retrieval and Com- puter Vision. Therefore, possible limitation of our work is its generalizability to other knowledge graphs. We hypothesize that our work is generaliz- able to other knowledge graphs, such as DBPedia, BabelNet, and Open Multilingual WordNet, among others, since entity names (or aliases) and entity descriptions (or definitions) are often available in many of them. Our hunch is partially demon- strated by our empirical experiments on Multilin- gual Knowledge Graph Completion (see Section 6), evaluate the impact of M-NTA on DPB-5L, which is constructed from DBPedia. However, we hope that our work will raise awareness on the is- sues of multilingual coverage and precision of tex- tual information on as many knowledge graphs as possible, and inspire future work to investigate the
M-NTA. In Section 5.2, we demonstrate that M- NTA is able to combine information from MT, ws, and LLMs, successfully outperforming the three approaches in increasing coverage and precision of entity names across the 10 languages of WikiKGE- 10. However, one of its main limitations comes from the fact that M-NTA requires the output from MT, ws, and LLMs. therefore, its inference time and computational cost is equal to the sum of its individual components run sequentially. Since we want a knowledge graph to contain the best textual information possible, we believe that the increase in performance +12% in terms of aver- age F1 score on coverage increase compared to the second best system; +22% on increasing precision jjustifies the additional time and compute required to run M-NTA. However, we look forward to novel methods that will be able obtain the same or even better results while drastically decreasing the computational requirements.

---

Acknowledgements
Soumen Chakrabarti, Harkanwar Singh. Shubham Lo- hiya, Prachi Jain, and Mausam 2022. Joint comple- tion and alignment of multilingual knowledge graphs. In Proceedings of the 2022 Conference on Empiri- cal Methods in Natural Language Processing, pages 11922-1 1938, Abu Dhabi, United Arab Emirates. As- sociation for Computational Linguisties.
This work would not have been possible without the invaluable feedback by and conversations with Behrang Mohit, Saloni Potdar, Farima Fatahi Bayat, Ronak Pradeep, and Revanth Gangi Reddy.
Xuclu Chen, Muhao Chen, Changjun Fan, Ankith Up- punda, Yizhou Sun, and Carlo Zaniolo. 2020a. Mul- tilingual knowledge graph completion via ensemble knowledge transfer. In Findings of the Association for Computational Linguistics: EMNLP 2020, pages 3227-3238, Online. Association for Computational Linguisties.
References
Oshin Agarwal, Heming Ge, Siamak Shakeri, and Rami AI-Rfou. 2021. Knowledge graph based synthetic corpus generation for knowledge-enhaneee eennneeed language model pre- vre-training. In Proceedings of the 2021 Con- ference of the North American Chapter of the Asso- ciation for Computational Linguistics: Human Lan- guage Technologies, pages 3554-3565 3565, Online. As- sociation for Computational Linguistics.
Yang Chen, Chao Jiang, Alan Ritter, and Wei Xu. 2022. Frustratingly easy label projection for cross- tross-lingual transfer. arXiv preprint arXiv:221 1,613.
Zhe Chen, Yuehan Wang, Bin Zhao, Jing Cheng, Xin Zhao, and Zongtao Duan. 2020b. Knowledge graph completion: A review. IEEE Access, 8:192435- 192456.
Junwei Bao, Nan Duan, Zhao Yan, Ming Zhou, and Tiejun Zhao. 2016. Constraint-based question an- swering with knowledge graph. Proceedings of COLING 2016, the 26th International Conference on Computational Linguistics: Technical Papers, pages 2503-2514, Osaka, Japan. The COLING 2016 Orga- nizing Committee.
Simone Conia and Roberto Navigli. 2021. Framing word sense disambiguation as multi-label problem for model-agnostic knowledge integration. In Pro- ceedings of the 16th Conference of the European Chapter of the Association for Computational Lin- guistics: Main Volume, pages 3269-3275 Online. Association for Computational Linguisties.
Edoardo Barba, Tommaso Pasini, and Roberto Nav- igli. 2021. ESC: Redesigning WSD with extractive sense comprehension. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 4661-4672, Online. Association for Computational Linguistics.
Alexis Conneau, Kartikay Khandelwal, Naman Goyal, Vishrav Chaudhary, Guillaume Wenzek, Francisco Guzmán, Edouard Grave, Myle Oit, Luke Zettle- moyer, and Veselin Stoyanov. 2020. Unsupervised cross lingual representation learning scale. In Pro- ceedings of the 58th Annual Meeting of the Asso- ciation for Computational Linguistics, pages 8440- 8451, Online. Association for Computational Lin- guistics.
Michele Bevilacqua and Roberto Navigli. 2020. Break- ing through the 80% glass ceiling: Raising the state of the art in word sense disambiguation by incorpo- rating knowledge graph information. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 2854-2864, On- line. Association for Computational Linguistics.
Marta R Costa-jussà, James Cross, Onur Çelebi, Maha Elbayad. Kenneth Heafield, Kevin Heffernan, Elahe Kalbassi, Janice Lam, Daniel Licht, Jean Maillard, et al. 2022. No language left behind: Scaling human-centered machine translation. arXiv preprint M:Xiv:2077.0462
Jan A. Botha, Zifei Shan, and Daniel Gillick. 2020. En- tity Linking in 100 Languages. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 7833-7845, Online. Association for Computational Linguistics.
Nicola De Cao, Ledell Wu, Kashyap Popat, Mikel Artetxe, Naman Goyal, Mikhail Piekhanov, Luke Zettlemoyer, Nicola Cancedda, Sebastian Riedel, and Fabio Petroni. 2022. Multilingual autoregressive en- tity linking. Transuctions of the Association for Com- putational Linguistics. 10:274-290.
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert- Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel Ziegler. Jeffrey Wu, Clemens Winter, Chris Hesse, Mark Chen, Eric Sigler, Ma- teusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, llya Sutskever. and Dario Amodei, 2020. Language models are few-shot learners. In Ad- vances in Neural Information Processing Systems, volume 33, pages 1877-1901. Curran Associates, Inc.
Jacob Devlin, Ming- Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language under- standing. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Tech- nologies, Volume (Long and Short Papers), pages 4171-4188 Minneapolis, Minnesota. Association for Computational Linguistics.

---

Qingyu Guo, Fuzhen Zhuang, Chuan Qin, Hengshu Zhu, Xing Xie, Hui Xiong, and Qing He. 2022. A survey knowledge graph-based recommender sys- tems. IEEE Transactions on Knowledge and Data Engineering, 34(8):3549-366
Shaoxiong Ji. Shirui Pan, Erik Cambria, Pekka Martti- nen, and Philip S. Yu. 2022. A survey on knowledge graphs: Representation, acquisition, and applications. JEEE Transactions on Neural Networks and Learning Systems, 33(2):494-514
Kelvin Guu, Kenton Lee, Zora Tung, Panupong Pasupat, and Mingwei Chang. 2020. Retrieval augmented language model pre-training. In P Proceedings of the 37th International Conference on Machine Learning. volume 119 of Proceedings of Machine Learning Research, pages 3929-3992 -938..93...10 PMLR.
Xin Ji and Wen Zhao. 2021. SKGSUM: Abstractive document summarization with semantic knowledge graphs. In 2021 International Joint Conference on Neural Networks (JJCNN), pages 1-8.
Ziwei Ji. Nayeon Lee, Rita Frieske. Tiezheng Yu, Dan Su, Yan Xu, Etsuko Ishii, Ye Jin Bang, Andrea Madotto, and Pascale Fung. 2023, Survey of halluci- nation in natural language generation. ACM Comput. Surv. 55(12).
Daniel Hershcovich, Stella Frank, Heather Lent, Miryam de Lhoneux, Mostafa Abdou, Stephanie Brandl, Emanuele Bugliarello, Laura Cabello Pi- queras, This Chalkidis, Ruixiang Cui, Constanza Fierro, Katerina Margatina, Phillip Rust, and Anders Søgaard. 2022. Challenges and strategies in cross- cultural NLP. In Proceedings of the 60th Annual Meeting of the Association for Computational Lin- guistics (Volume I: Long Papers), pages 6997-7013, Dublin, Ireland. Association for Computational Lin- guistics.
Jens Lehmann, Robert Isele, Max Jakob, Anja Jentzsch, Dimitris Kontokostas. Pablo N. Mendes, Sebastian Hellmann, Mohamed Morsey, Patrick van Kleef, Sören Auer, and Christian Bizer. 2015. Dbpedia- A large-scale, multilingual knowledge base extracted from wikipedia. Semantic Web, 6(2):167-195
Johannes Hoffart, Fabian M. Suchanek, Klaus Gerard de Melo,
Mike Lewis. Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, AbdeIrahman Mohamed, Omer Levy. Veselin Stoyanov, and Luke Zeullemoyer. 2020. BART: Denoising sequence-to-sequenee pre-training for natural language generation, translation, and com- prehension. In Proceedings of the 58th Annual Meet- ing of the Association for Computational Linguistics, pages 7871-7880, Online. Association for Computa- tional Linguistics.
Berberich, Edwin and Gerhard Weikum. 2011. YAGO2: exploring and querying world knowledge in time, space, context, and many languages. In Proceedings of the 20th In- ternational Conference World Wide Web, WWW 2011, Hyderabad, India, March 28- April 1, 2011 (Companion Volume). pages 229-232. ACM.
Aidan Hogan, Eva Blomqvist, Michael Cochez, Clau- dia D' D'amato, Gerard De Melo, Claudio Gutierrez, Sabrina Kirrane, José Emilio Labra Gayo, Roberto Navigli, Sehastian Neumaier, Axel-Cyrille Ngonga Ngomo, Axel Polleres. Sabbir M. Rashid, Anisa Rula, Lukas Schmelzeisen, Juan Sequeda, Steffen Staab, and Antoine Zimmermann. 2021. Knowledge graphs. ACM Comput. Surv. 54(4).
Shuyang Li, Mukund Sridhar, Chandana Satya Prakash, Jin Cao, Wael Hamza, and Julian McAuley. 2022a. Instilling type knowledge in language models via multi-task QA. In Findings of the Association for Computational Linguistics: NAACL 2022, pages 594- 603, Seattle, United States. Association for Compu- tational Linguistics.
Luyang Huang, Lingfei Wu, and Lu Wang. 2020. Knowledge graph-augmented abstractive summariza- tion with semantic-driven cloze reward. In Proceed- ings of the 58th Annual Meeting of the Association for Computational Linguistics. ACL 2020, Online, July 5-10, 2020, pages 1094-5107 Association for Computational I. Linguistics.
Zhuliu Li, Yiming Wang, Xiao Yan, Weizhi Meng. Yanen Li, and Jaewon Yang. 2022b. Taxotrans: Taxonomy- guided entity translation. In Proceedings of the 28th ACM SIGKDD Conference on Knowl- edge Discovery and Data Mining, KDD '22, page 3279-3287, New York, NY, USA. Association for Computing Machinery.
Xiao Huang, Jingyuan Zhang, Dingcheng Li, and Ping Li. 2019. Knowledge graph embedding based ques- tion answering. In Proceedings of the Twelfth ACM International Conference on Web Search and Data Mining, WSDM 19, page 105-113, New York, NY, USA. Association for Computing Machinery.
Yankai Lin, Zhiyuan Liu, Maosong Sun, Yang Liu, and Xuan Zhu. 2015. Learning entity and relation em- beddings for knowledge graph completion. In Pro- ceedings of the Twenty-Ninah AAAI Conference on Artificial Intelligence, January 25-30, 2015, Austin, Texas, USA, pages 2181 2187. AAAI Press.
Zijie Huang. Zheng Li, Haoming Jiang, Tianyu Cao, Hanqing Lu, Bing Yin, Karthik Subbian, Yizhou Sun, and Wei Wang. 2022. Multilingual knowledge graph completion with self- supervised adaptive graph align- ment. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Vol- ume I: Long Papers). pages 474 485, Dublin, Ire- land. Association for Computational Linguistics.
Linlin Liu, Xin Li, Ruidan Hc, Lidong Bing, Shafiq Joty, and Luo Si. 2022. Enhancing multilingual lan- guage model with massive multilingual knowledge triples. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 6878-6890, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics.

---

Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Man- dar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. RoBERTa: A robustly optimized BERT pretraining approach. arXiv preprint 1xXiv1907.11192
Ciyuan Peng, Feng Xia. Mehdi Naseriparsa, and Francesco Osborn.. 2023b. Knowledge graphs: Op- portunities and challenges. Artificial Intelligence Review.
Fabio Petroni, Tim Rocktäschel. Sebastian Riedel, Patrick Lewis, Anton Bakhtin, Yuxiang Wu, and Alexander Miller. 2019. Language models as knowl- edge bases? In Proceedings of the 2019 Confer- ence on Empirical Methods in Natural Language Pro- cessing and the 9th International Joint Conference Natural Language Processing (EMNLP-IJCNLP), pages 2463 2473, Hong Kong, China, Association for Computational Linguistics.
Shayne Longpre, Lu, and Joachim Daiber. 2021. MKQA: A linguistically diverse benchmark for mul- tilingual open domain question answering. Transac- tions of the Association for Computational Linguis- tics, 9:1389-1406.
Kenneth Marino, Ruslan Salakhutdinov, and Abhinav Gupta. 2017. The more you know: Using knowledge graphs for image classification. In 2017 IEEE Con- ference on Compuler Vision and Pattern Recognition (CVPR), pages 20-28.
Luigi Procopio, Simone Conia, Edoardo Barba, and Roberto Navigli. 2023. Entity disambiguation with entity definitions. In Proceedings of the 17th Con- ference of the European Chapter of the Association for Computational Linguistics, pages 1297-1303, Dubrovnik, Croatia. Association for Computational Linguistics.
Marco Maru, Simone Conia. Michele Bevilacqua, and Roberto Navigli. 2022. Nibbling at the hard core of Word Sense Disambiguation. In Proceedings of the 60th Annual Meeting of the Association for Compu- lational Linguistics (Volume 1: Long Papers), pages 4724 4737, Dublin, Ireland. Association for Compu- tational Linguistics.
Alec Radford, Karthik Narasimhan, Tim Salimans, and Ilya Sutskever. 2019. Improving language under- standing by generative pre-training.
Niklas Muennighoff, Thomas Wang, Lintang Sutawika, Adam Roberts, Stella Biderman, Teven Le Scao, M Saiful Bari, Sheng Shen, Zheng-Xin Yong, Hai- ley Schoelkopf, Xiangru Tang. Dragomir Radev, Al- ham Fikri Aji, Khalid Almubarak, Samuel Albanie, Zaid Alyafeai, Albert Webson, Edward Raff, and Colin Raffel. 2022. Crosslingual generalization through multitask finetuning.
Jonathan Raiman and Olivier Raiman. 2018. DeepType: Multilingual entity linking by neural type system evolution. In Proceedings of the Thirty-Second AAAI Conference on Artificial Intelligence, (AAAI-18). the 30th innovative Applications of Artificial Intelligence (IAAI-18), and the Sth AAAI Symposium on Éduca- tional Advances in Artificial Intelligence (EAAI-18), New Orleans, Louisiana, USA, February 2-7, 2018, pages $406-5413. AAAI Press.
Roberto Navigli, Michele Bevilacqua, Simone Conia, Dario Montagnimi, and Francesco Cecconi. 2021. Ten years of babelnet: A survey. In Proceedings of the Thirtieth International Joint Conference on Artificial Intelligence, IJCAI 2021, Virtual Event / Montreal, Canada, 19-27 August 2021, pages 4559- 4567. IJCAL
Ridho Reinanda, Edgar Meij, and Maarte de Rijke. 2020. Knowledge graphs: An information retrieval perspec- tive. Foundations and Trends@ in Information Re- trieval, 14(4):289 -444
Teven Le Scao, Angela Fan, Christopher Akiki, EI- lie Pavlick, Suzana Ilié, Daniel Hesslow, Roman Castagné, Alexandra Susha Luccioni, François Yvon, Matthias Gallé, Jonathan Tow, Alexander M. Rush, Stella Biderman, Alhert Webson, Pawan Sasanka Am- manamanchi, Thomas Wang, Benoît Sagot, Niklas Muennighoff, Albert Villanova del Moral, ... and Thomas Wolf. 2023. BLOOM: A paramterre open-access multilingual language model.
Roberto Navigli, Simone Conia, and Bjöm Ross. 2023, Biases in large language models: Origins, inventory and discussion. J. Data and Information Quality.
Maximilian Nickel, Kevin Murphy. Volker Tresp, and Evgeniy Gabrilovich. 2016. A review of relational machine learning for knowledge graphs. Proceedings the IEEE, 104(1):11-33.
Laurel J. Orr, Megan Leszezynski, Neel Guha, Sen Wu,
Phillip Schneider, Tim Schopf. Juraj Vladika, Mikhail Galkin, Elena Simperl, and Florian Matthes. 2022. A decade of knowledge graphs in natural language processing: A survey. In Proceedings of the 2nd Conference of the Paccffic Chapter of the Asso- ciation for Computational Linguistics and the 12th International Joint Conference on Natural Language Processing ( (Volume I: Long Papers), pages 601-614, Online only. Association for Computational Linguis- tics.
Simran Arora, Xiao Ling, and Christopher Ré. 2021. Bootleg: Chasing the tail with self-supervised named entity disambiguation. In 1 11th Conference on In- novative Data Systems Research, CIDR 2021, Vir- tual Event, January 11-15, 2021, Online Proceedings. www.cidrdb.org.
Baolin Peng. Michel Galley, Pengcheng He, Hao Cheng. Yujia Xie, Yu Hu, Qiuyuan Huang, Lars Liden, Zhou Yu, Weizhu Chen, and Jianfeng Gao. 2023a. Check your facts and try again: Improving large language models with external knowledge and automated feed- back. In arXiv.
Baoxu Shi and Tim Weninger. 2018. Open-world knowl- edge graph completion. In Proceedings of the Thirty- Second AAAI Conference on Artificial Intelligence,

---

(AAAI-18), the 30th innovative Applications of Arti- ficial Intelligence (IAAI-18), and the 8th AAAI Sym- posium Educational Advances in Artificial Intel- ligence (EAAI-18), New Orleans, Louisiana, USA, February 2-7, 2018, pages 1957-1964. AAAI Press.
A Creating WikiKGE-10
In this section, we provide more details on the creation process of WikiKGE-10, our novel human- curated dataset for evaluating automatic approaches on KGE of Wikidata entity names.
Ralf Steinberger, Bruno Pouliquen, Mijail Kabadjov, Jenya Belyaeva, and Erik van der Goot. 2011. JRC- NAMES: A freely available, highly multilingual named entity resource. In Proceedings of the In- ternational Conference Recent Advances in Natural Language Processing 2011, pages 104-110, 10, Hissar, Bulgaria, Association for Computational Linguistics.
A.1 Choice of languages
As mentioned in Section 3.4, one of the main de- sign decision for our benchmark is the selection of 10 languages from set of diverse typologically- different linguistic families:
Denny Vrandečić and Markus Krötzsch, 2014. Wiki- data: A free collaborative knowledgebase. Commun. ACM, 57(10):78-85.
West Germanic: English, German;
Romance: Spanish, French, Italian;
Quan Wang, Zhendong Mao, Bin Wang. and Li Guo. 2017. Knowledge graph embedding: A survey of approaches and applications. IEEE Transactions on Knowledge and Data Engineering, 29(12)2722 2743.
Semitic: Arabic;
Sino-Tibetan: Chinese (simplified);
Slavic: Russian;
Wenhan Xiong, Jingfei Du, William Yang Wang, and Veselin Stoyanov. 2020. Pretrained encyclopedia: Weakly supervised bhowledge-pretraiing model. In 8th International Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia. April 26-30, 2020. OpenReview.net.
Koreanic: Korean;
language
Japonie: Japanese.
This design choice makes WikiKGE-10 10 challeng- ing, as the set of symbols used in each language may or may not vary significantly: for example, a person name may be the same in English and French, but it is highly unlikely that a person name is written in the same way in English and Chi- nese, which requires at least transliteration. More- over, the transliteration process between English and Chinese (and also other languages, such Japanese) is not always deterministic, making it difficult to rely on rule-based approaches to trans- late a name between these two distant languages. We focused on languages that can be considered high/medium--resouree as our quantitative analysis in Section 3.2 shows that coverage of textual in- formation is still far from ideal even for the most popular entities (top- (top-10%) of those high/medium- resource languages. We leave the expansion of our benchmark to lower-resource languages future work.
Yan Xu, Mahdi Namazifar, Devamanyu Hazarika, Aish- warya Padmakumar, Yang Liu, and Dilek Hakkani- Tür. 2023. KILM: Knowledge injection into encoder-decoder language models. arXiv preprint arXiv:2302.09170.
Linting Xue, Noah Constant, Adam Roherts, Mihir Kale, Rami Al-Rfou, Aditya Siddhant, Aditya Barua, and Colin Raffel. 2021. mt:: massively multilingual pre-trained text-to-text transformer. In Proceedings of the 2021 Conference of the North American Chap- ter of the Association for Computational Linguistics: Human Language Technologies, NAACL-IILI 2021, Online, June 6-11, 2021, pages 483-498. Association for Computational Linguistics.
Daochen Zha, Zaid Pervaiz Bhat, Kwei-Herng Lai, Fan Yang, Zhimeng Jiang, Shaochen Zhong, and Xia Hu. 2023. Data-centric artificial intelligence: A survey.
Yuyu Zhang, Hanjun Dai, Zornitsa Kozareva, Alexan- der J. Smola, and Le Song. 2018. Variational reason- ing for question answering with knowledge graph. In Proceedings of the Thirry-Second AAAI Conference on Artificial Intelligence, (AAAI-18), the 30th inno- vative Applications of Artificial Intelligence (IAAI- 18), and the 8th AAAI Symposium on Educational Advances in Artificial Intelligence (EAAI-18), New Orleans, Louisiana, USA, February 2-7, 2018, pages 6069-6076. AAAI Press.
A.2 Human annotation process
The objective of the annotation process to sug- gest and rate entity names in a target language. First, given an entity, the human annotators were asked to familiarize themselves with its informa- tion: the user interface for the task provided the

---

The Sow me le
Gougl
caber cubdiies
Rabor Mombo
soul recenalle
Zim
VisinssA
WINIFETAA
Forre India
Force Incia
Coprr re crome cono inuetin lew ollen
1991
hoia herui sever
encire nemer oppnngg
kiee Covee eln tome/haa nerwerr
Question
Entity
Marodae lunaae rudall
UT used for the annotation task: the annotators coudl familiarize themselves with the task with outline of the task instructions (detailed guidelines could be read in separate page) and the information about the entity, including its names English and its Wikipedia pages English the target language (Italian in this case).
Figure 2:
entity names and a short description of the given en- tity in English retrieved from Wikidata, well as built-in panel that directly displayed side-by-siae the Wikipedia articles of the corresponding entity both in English and in the target language, if avail- able. This allowed human annotators to familiarize themselves with the entity and catch commonalities and differences between English and non-English information at glance without leaving the annota- tion tool.
2 Spelling issues. The name contains minor issues, for example, spelling errors missing digits. For example, "Michael Jacson" (notice the missing "k") should not be used refer "Michael Jackson".
3 Generic, rare or incomplete. The name can be used to refer to this entity but it is very generic, rare or incomplete. For exam- ple, "Barack" can be used refer to "Barack Obama" or "game" "video game." Note that nicknames or stage names like "Air Jordan" for Michael Jordan (basketball player) or "Money" for Floyd May- weather (boxer) do not fall into this category; they should be categorized as "good fit" (see below).
After learning about the entity, the annotators were tasked with rating entity names that are valid for the given entity with respect to the target lan- guage, i.e., if an entity name is valid only in lan- guages that are different from the target language of interest, the annotators were explicitly asked to categorize such names as invalid. More specifically, for each name, an annotator could choose one of the following options:
be used to refer
4 Good fit. The name is good way refer to this entity (for example, one of its common names, nickname, or an acronym). For example, "Harvard" can be used to refer to "Harvard University", "WB" can be used to refer to "Warner Bros.", ", "Schumi" is valid nickname for "Michael Schumacher".
The name should not be used to refer to the entity in the target language. For example, "pomodori marci" - the literal translation of "tomatoes that are rotten (fruit)" in Italian should never be used to refer to Rotten Tomatoes (the media review site). In addition, the name should always be valid in the target locale; therefore, a name another language that is not recognized the target locale should be considered incorrect.
Incorrect.
5 Perfect fit. The name is the most appro- priate name for this entity (usually, its most common name). For example, "Harvard Uni- versity" (instead of just "Harvard"), "Barack

---

Tryhht "Fore Inda's
Nowe walpla spte bo Coag
spee uere arw Lil
Q paction 2: Do you krow addttoral for Force India' iubllee?
Q Queedon èl Faedbock
UI used for the annotation task: the annotator had to rate names from to 5. Before providing rating. they can easily double-check the name in consideration on Web search engine. As shown this figure, the annotators could also suggest new names, which will inserted in the pool of entity names to grade.
Figure 3:
Obama" (instead of "Barack Hussein Obama II"). In other words, it is the most common or popular entity name to reference the intended entity.
they knew about other possible valid names. Each suggested name was inserted in the pool of entity names to validate, and, therefore, graded by 3 an- notators. On the contrary, annotators could not suggest invalid entity names for an entity, as our objective was to focus on the errors that are already in Wikidata, but could provide feedback case they noticed that something was wrong in the task.
Annotators were given the choice opt out from rating an entity name in case they deemed they did not have enough context (e.g., information from the Wikipedia pages the entities in English and
target language) or they did not feel knowledgeable enough about the topic.
A.3 Quality assurance and inter-annotator agreement
Before confirming their selection, each annotator had to double-check their choice by searching exact matches of the name under consideration using Web search engine; UI component allowed the annotators directly look up for exact matches in the target language without manually typing a query, making the search easier and speeding up the annotation process. Forcing the annotators to take this extra step allowed them to verify that named they deemed invalid was indeed invalid, i.c., or few results from the search engine, not associated to the entity of interest. An example of an annotation task is shown in Figures and 3.
To guarantee a high-quality output, before partic- ipating to the annotation process, each human an- notator had to pass an entrance test, which con- sisted in studying set of guidelines which in- troduced the annotator to the concepts of entities and knowledge graphs, described the task and the UI elements, and provided a few examples with illustrations and in rating 50 entity names cor- rectly. Annotators that could not pass the entrance test could not participate to the actual annotation process (we did not use the 50 entity names in the entrance test in the final dataset).
We note that annotators could also suggest new names the target language for each entity
For each target language, we only hired annota- tors that could certify their proficiency in English

---

pendorff's s alpha, which shows strong agreement with an average of 0.96 across all languages, we can see in Figure 5. Overall, the strong inter- annotator agreement scores validate the results of the annotation process.
0.80
00.78
0.81
0.79
0.79
0.78
Shong agraram
B Related Work: Addendum
While WikiKGE- 10 is the first benchmark designed to aid development and evaluation of systems for in- creasing coverage and precision of entity names in multilingual knowledge graphs, there has been pre- vious work that tried address this issue in other ways. Among them, we acknowledge the existence of JRC-Names (Steinberger et al., 2011). Here, we provide more details on the fundamental dif- ferences WikiKGE- 10 and JRC-Names, including: i) WikiKGE-10 is completely manually-creater ii) WikiKGE-10 is mapped 1 1-to-1 Wikidata; iii) WikiKGE-10 is not limited persons and orga- nizations; iv) JRC-Names considers names with spelling mistakes as valid names (as they may ap- pear in real-life scenarios), whereas WikiKGE-ID 10 considers them incorrect (as our objective is to ob- tain a multilingual knowledge graph that is as clean as possible); v) JRC-Names does not distinguish between entities that have the same name, since it is "very likely that different persons sharing the same first and last name have the same identifier because no disambiguation mechanism is place." "
Figure 4: Pairwise inter-annotator agreement measured with Cohen's 's Kappa shows strong agreement at the end of the annotation process.
Arppendorta alphe
1.00
1185
0.85
0.85
0.96
0.05
0.90 0:96 0.96
Strong
Tectatinn
201
22
30
80 2H
Figure 5: Inter-annotator agreement measured with Krippendorff's alpha, which takes into account the dinality of the ratings (1-5), shows strong agreement among annotators.
and the target language. Annotators were compen- sated according to the standard hourly wages of their geographic location. On average, each an- notator spent about minute for rating an entity name and about 5 minutes on each entity. Since each entity name was rated by 3 annotators, we estimate that the total human time required by the annotation process is 3 annotators 10,000 entities minutes / 60 minutes 2,500 hours.
C Methodology: Addendum
In this section, we provide more details on the methods we investigate in our paper, namely, MT, WS, LLMs, and M-NTA.
C.1 Contextualizing entity names
As mentioned in Section 4.1, converting entity names from one language to another - by using ma- chine translation, looking them up with Web search engines, querying language models is chal- lenging because entity names can be ambiguous. Therefore, we contextualize entity names before converting them from one language to another lan- guage, i.e., we add information that a system can use to disambiguate an entity name and produce the correct output in the target language.
At the end of the annotation process, we mea- sured the inter-annotator agreement in two ways. First, we computed pairwise inter-annotator agree- ment using Cohen's kappa. As shown Fig- ure 4, we can observe an average agreement of 0.79, where score of 0.60 is usually considered to represent substantial agreement and 0.80 is usu- ally regarded as strong agreement. We also stress that Cohen's 's kappa does not take into account the cardinality of the rating values, i.e., for Cohen's kappa there is no difference between a 1-vs-5 and 4-vs-5 disagreement. Therefore, we also measured the overall inter-annotator agreement using Krip-
More specifically, given the fact that we already know the entity identifier associated to the entity
'The original paper on Krippendorlf's 's alpha suggests that tentative conclusions can be made with a score greater than 0.67 and strong conclusions can be made with a score greater than 0.80.

---

name we would like to translate, we retrieve its cor- responding description from Wikidata in the same language as the entity name, and use it to form a pseudo-natural language sentence." 6 For example, the entity name Apple is contextualized as "Apple is an American technology company" and "Apple is a fruit of the apple tree" depending on whether it corresponds to entity Q312 or Q89, respectively. In case of missing entity descriptions for target language, we construct a simple entity description starting from its instance-of statements in Wiki- data, c.g., "Albert Einstein a human." " While more complex strategies or more relations may be used to better contextualize entity names, devis- ing more complex strategies which may require separate ad hoc solutions for MT, WS, and LLMs is beyond the scope of this paper. We leave the investigation of more complex techniques for entity name contextualization to future work.
subset of manually-inspected instances. While this alignment system can be replaced by more com- plex alignment techniques, our analysis suggests that alignment errors are not the primary factor in end-to-end evaluation; we measured the number of errors attributable to misalignments and found that only 2% of the translated sentences contains such errors. Therefore, we can conclude that alignment errors are not major bottleneck end-to-end per- formance on WikiKGE- 10 probably due the simplicity of the syntactic structure of the sentences that result from the contextualization process and leave the investigation of more complex alignment systems to future work.
C.3 MT: implementation details
In our experiments with MT, we decided to limit the number of source languages to 7, namely, Ger- man, English, Spanish, French, Italian, Japanese, and Chinese. The main reason behind this choice is that the quality translations from automatic sys- tems has been shown to still lag behind when the source language is a lower-resource language, e.g.. Korean. Therefore, in this work, we focus our at- tention on higher-resouree languages for which MT has been proven to achieve satisfactory results on several standard benchmarks, allowing us to iterate faster. We hypothesize that translating from lower- resource languages does not result in performance that significantly better than what we can see in Table 2, even when the linguistic families of the source and target languages are close. However, we leave an investigation on the effect of carefully choosing source-target language pairs for MT to future work.
C.2 Aligning and de-contextualizzng entity names
While the advantage of contextualizing entity names is evident, the main disadvantage is that system will "translate" an entity name and also its contextualization information, possibly mixing the two types of textual information. This issue target
is particularly relevant when translating language with a syntax that is significantly different from the source language or to a target language with non-trivial segmentation rules, e.g., from En- glish to Japanese or Chinese. Therefore, we need to de ce-contexttullze the translated name, i.e., we need to align the translated name to the original name and remove the contextualization information that was translated together with the name.
C.4 WS: implementation details
To address this issue, we follow recent stud- ies (Chen et al., 2022) in alignment techniques, which show that MT is surprisingly robust to the insertion of symbols in the input sentence. More specifically, we indicate the start and the end of the entity name in the input sentence with special mark- ers; for example, "[Apple] is an American technol- ogy company." After translating the contextualized entity name into the target language, we detect the start and end markers in the translation and use their position to extract the translated entity name. Our analysis reveals that such an alignment system produces valid alignments most of the time in a
In Section 4.1, we discussed how WS can be used to retrieve entity names in target language: given an entity name in a source language (, we can per- form a search using a query like "[entity-name) in [4J" to obtain results in a target language 4, More- over, we can enrich the query by adding contextual information in the form of Wikidata descriptions, as discussed in section C.I, resulting in enriched queries like "/entity-name] (fentity- descriit-dds in [4,]" to mitigate the problem of ambiguous names, e.g., not only there are more than 10 people in Wikipedia that could be referred to Michael Jor- dan but also songs and movies.
"Wikidata descriptions can be retrieved from the Wikidata dump. Each entity may have multiple Wikidata descriptions, one for each language if available.
More specifically, given an entity e and one of its names ey and its Wikidata description eg in a

---

source language 4. build search query as de- scribed above, limiting the choice of to English. Then, we parse the HTML response and collect the most frequently highlighted terms, i.e., those terms that are in bold (between <h></b> tags) or empha- sized (between <em></em> tags), in the top- 10 websites returned by the search engine. Finally, we keep the top-5 entity names retrieved from the col- lected terms if they appear least times among the highlighted results. As discussed in Section 5.2, such an approach even though it tries to imitate how humans look up information on the Web re- sults in a significant amount of noise due to the collection of significant number of terms that are only aaaanticalll-relttee to the query and not semantic matches.
have the same entity type, e.g., if want generale the name for and is person, then also
the example entity é shall be a person. Notwith- standing the input/output example provided, we observe that sometimes LLMs, even when they
output correct names, do not conform to the same input/output format as the example, e.g.. they add preambles ("the name of X is Y", "as a language model, I.."") explanations ("X because..."). This makes hard to extract the relevant portion of text, resulting in alignment errors.
C.6 M-NTA: implementation details
In this section, we provide more details on three im- portant factors for the implementation of M-NTA. namely, the value of X, the choice of é, and the individual contribution of each sub-system (MT, WS, and LLMs) in M-NTA.
C.5 LLMs: implementation details
In our experiments, investigate two main LLMs: mTO and GPT. The former is the instruction- finetuned version of mT5, state-of-the-ar multi- lingual LLM. For mT5, we take into account three variants- - large, xl, and xxl - which differ in their size to investigate if and to what extent increasing the number of trainable parameters in language model is beneficial for the task under consideration. evaluate the effective-
C.6.1 The value of X
In Section 4.2, we introduced M-NTA, our novel approach combine MT, WS, and LLMs, and described how it scores and ranks the answers Y = {y aty) 2 }} according to threshold hyperparameter X, mentioning that most robust choice for coverage. Here, we expand our discussion on A, showing how the choice of its value can significantly vary the precision and recall of the answers provided by M-NTA.
the
For our experiments, ness of m55 and GPT with one-shot prompts, i.c., we provide a description of the task and one ex- ample of input/output the LLM before requiring them to generate the entity name of interest. More specifically, each prompt is constructed as follows:
At a high level, the intuition behind À is that it is hyperparameter that controls the number of "supporting evidences" required by M-NTA con- sider an answer as plausible; on the contrary, if an answer is supported by fewer than À evidences, then M-NTA considers such an answer as noise. Therefore, we can expect that increasing the value of À will result in more precise predictions at the cost of recall, and decreasing the value of À will result in more broad coverage but also less precise answers. This indeed the case in our experiments, as we can see in Figures and 7, in which we can observe that increasing the value of À decreases the overall recall while increasing the precision of the answers on a sample of the Italian and Korean test sets of WikiKGE- Given the results of M-NTA for different values of À across the 10 languages of WikiKGE-10,
Task definition: given an entity name in En- glish and a short description of the entity in English, complete the following with the cor- responding entity name in [t.).
Example:
English name: [ē]]
English description: [eg]
- [4]] name: [é]]]
Task:
English name: [e[]
English description: [eg]
observed that À 2 is em- pirically the best choice on average if we want balance precision and recall in coverage. However, we also note that the decision about the value of can be also affected by the downstream applica- tion of interest: the use case is adding textual
- [1]] name:
where 4 is the target language, is the entity used for the example, and e the entity of interest. We choose the example entity è at random from the 10% entities with the only constraint that and

---

IT - Recall vs.
IT - Precision vs.
R
P
F1
100%
95.5
96.9 98.4
M-NTA
89.7
42.4
57.1
83.7
82.9
M-NTA 2-2
71.4
81.5
75.6
Rocall Prezision (N)
M-NTA, A=3
56.4
82.3
66.5
51.5
M-NTA, A=4
41.9
87.3
56.0
47.1
M-NTA 55
23.2
90.3
35.5
M-NTAA=6
10.4
93.8
18.0
20.1
Table 5: Recall (R), Precision (P), and Fl Score of M-NTA with different values of M-NTA across the 10 languages of WikiKGE-I0 The value in M-NTA can be tuned to have broad recall high precision.
096
Value of
Value of
Figure 6: Recall and Precision (%) of M-NTA in the Italian test set WikiKGE-10 (coverage) for increasing values of X, ranging from 1 to 6. We can observe how the Recall decreases as the Precision increases.
C
P
M-NTAFull
53.6
79.9
M-NTA no-WA
53.2
79.8
KO Recall vs.
KO Precision vs.
M-NTAmo-LLMM
43.8
71.1
100%
96.1
M--TTAnoWShho-LLMM
43.2
70.9
86.7
87.3
91.0
81.9
Table 6: Ablation study the individual components of M-NTA on coverage (C) and precision (P). All re- sults reported for M-NTA with À and for precision.
72.9
Recal Precision
63.6
53.7
43.9
40.1
on entity names, for which even a slight variation between two names can mark the difference be- tween correct name and an incorrect one, c.g., Olivier and Oliver. Therefore, we choose exact match between lower-cased, punctuation-strippe entity names as the function Φ, i.e., name y is supported by another name y' if and only if y'. except for letter casing (e.g., Canary and canary) and punctuation (Michael B Jordan and Michael B. Jordan). As we will see in section C.7, other forms of á may be more appropriate for types of textual information different from entity names.
09%
4
2
5
Valuee of
Value A
Figure 7: Recall and Precision (%) of M-NTA in the Korean test set of WikiKGE-10 (coverage) for increas- ing values X, ranging from 1 6. We can notice the same trend as in Figure 6.
information to knowledge graph for direct user consumption, then may want to prefer precision over recall and increase the value of À accordingly; otherwise, if we want use textual information for the creation of multilingual embeddings, then we may more interested in recall for covering as many entities as possible.
C.6.3 Ablation study
Throughout the paper, we mentioned multiple times that the main strength of M-NTA is its capability to combine MT, WS, and LLMs. Here, we carry out an ablation study to quantify and better understand the individual impact of each subsystem in M- NTA. More specifically, we compare the results of the "full" M-NTA to M-NTA without Google Web Search (M-NTAm--ws) without GPT-3.5 (M- NTAmo-LLM). and only with MT from languages M.-NTAAno-SSmooLLLMM As we can see in Table 6, even when M-NTA does not rely on answers from WS and LLMs, the results of M-NTA WS.
answers provided by
C.6.2 The choice of
One important factor in the design of M-NTA is the choice of the function @'v,v/) - (0, 1), which establishes whether answer y from system f(-) is supported by the answer y/ from another system f'(.). While & can be any "similarity" met- ric, e.g., a measure of vector similarity, the final choice depends on the type of textual information represented by each answer. this paper, we focus

---

Entity
Source
Entity description
Bufuri
Wikidata
"company"
Q1002164
M-NTA
"car manufacturer"
Reinhard Zöllner
Wikidata
"German historian"
Q100562
M-NTA
"university professor and German historian"
Haukadalur
Wikidata
"valley"
Q1034430
M-NTA
"valley in Iceland with a geothermal area"
Paola Cortellesi
Wikidata
"Italian actress and singer"
Q1842721
M-NTA
"Italian actress, screenwriter, television author, comedian and singer (1973-)"
Washuzan Highland
Wikidata
"Japanese amusement park in Okayama prefecture"
Q10345485
M-NTA
"An amusement park in Shimotsui, Kurashiki City, Okayama Prefecture"
Table Selected examples of descriptions generated by M-NTA compared Wikidata. As you see, M-NTA is able to improve descriptions in high-resource languages. As we can see in this Table, M-NTA is able to provide important information in the description.
are better than simple translation from the best source language (English). This empirically vali- dates our hypothesis, i.e., different languages hold complementary knowledge and M-NTA able to combine such knowledge in an effective way.
i.e., designing an appropriate function é(v, v/) {0, }} (see section 4.2) to establish when descrip- tion y' = ot counts supporting evidence for a different description y eç. Indeed, description may imply another description even if they are not exact matches. For example, the English descrip- tion for Earth (Q2) is "third planet from the Sun in the Solar System", which implies the Spanish description "planet in the Solar System, third by distance from the Sun" (translated in English from Spanish). To address this issue, define & follows:
C.7 Applying M-NTA to entity descriptions
While the focus of WikiKGE- 10 is on entity names, the approaches described in Section 4.1 - MT, WS, and LLMs and M-NTA can also be applied to other types of textual information. As discussed in sections 2 and 3, entity descriptions are another popular type of textual information used in recent approaches. this section, we describe how MT and M-NTA can be easily adapted to convert entity descriptions from one language to another, while we leave a more in-depth study about the effective- ness of WS and LLMs for entity descriptions to future work.
((v.V' )(v-))= - {
if sim 1/) > 0.5 if sim y') - 0.5
where sim(-) is the cosine similarity between the vector representations of y and v/- We compute the vector representations of the descriptions by using XLMMRRBERTA (base) (Conneau et al., 2020).
Adapting the MT-based approach to generate en- tity descriptions in a target language is straightfor- ward. In section C.2, we discussed the necessity of using special markers to facilitate the extraction of the translated entity name from the translated sen- tence, e.g.. "[Apple] is an American multinational technology company." To extract the entity descrip- tion instead of the entity name, we can simply place the special markers around the entity description, e.g.. "Apple is an [American multinational technol- ogy company]." Thanks to this simple modification, the rest of the pipeline for the MT- approach can remain the same.
D WikiKGE-1k 10: Additional Results
In this section, we provide additional results to com- plement the main results described in Section 5. Indeed, it is interesting to observe how the re- sults would change if we slightly relax the metrics of coverage and precision. In particular, we relax coverage to provide positive score in case sys- tem is able to provide at least one valid entity name for a given entity. Similarly, we relax precision to provide positive score in case system is able to identify at least one invalid entity name for a given entity. Table provides an overview of the results. As one could expect, the scores on cov- erage increase, as it is easier to provide one valid
Adapting M-NTA to generate entity descriptions in a target language requires an additional step,

---

AR
DE
EN
FR
Arg
C
C
C
C
C
C
C
C
C
12.7
59.5
EN
714
49.9
76.6
61.7
51.9
53.3
74.8
52.6
78.9
59.5
51.5
38.5
70.5
54.1
67.8
47.6
55.5
42.7
09.9
51.0
MT From:
E5
57.5
32.9
59.8
38.9
57.7
33.2
66.3
39.9
68.6
43.7
61.1
37.2
56.2
30.7
26.4
57.5
34.5
61.1
67.0
63.1
71.9
47.8
26.6
62.8
99.2
29.9
25.7
61.4
34.4
IT
58.9
28.4
64.9
37.5
59.1
30.5
70.2
31.8
66.9
34.3
43.5
20.8
59.9
50.8
58.7
26.6
46.4
21.8
58.7
29.2
37.8
2H
38.0
16.5
27.6
9.6
2013
6.4
30.2
9.0
30.2
9.7
31.2
10.5
33.1
17.6
47.7
27.3
50.6
9.8
32.0
12.9
Googlesaanh
52.2
41.4
58.8
55.1
60.9
47.2
58.2
46.4
66,
58.2
34.2
22.2
45.3
57.7
42.1
42.7
39.4
31,8
51.8
42.5
m30
49.1
70.3
39,1
37.2
mT0m
53.2
38.1
57.2
46.6
71.8
45.2
61.0
14.6
65.3
56.1
36.1
19.2
66.4
38.8
46.0
38.6
39.5
32.0
53.6
40.1
12Mc
74.4
GPT:3.5
67.1
48.8
75.9
50.3
57.6
77.1
54.2
76.4
57.3
54.4
41.0
73.3
52.2
69.1
44.4
57.2
44.1
70.7
31.2
MENTA
75,9
70.6
79.7
73.1
67.8
S8.1
36.2
69.1
83.2
66.3
87.1
74.2
59.6
51.9
79.1
75.5
75.2
64.2
62.7
60.5
75.6
66.3
Table 8: Fl scores on entity names coverage (C) and precision (P) in WikiKGE-10 when identifying at least one valid name for coverage and at least one invalid name for precision. The symbol "_" is used to indicate that source and target languages are the same. Best results in bold.
Entities
Names voverage
coverage (%)
+00.0%
Reduction answerable queries (%6
-52.0%
+M-NTA
+M-NTA
DE
94.63
97.42
2.79
95.12
96.45
+1.33
45.0
+41.9%
EN
99.09
99.23
0.14
93.48
93.88
+0.40
+38.9%
ES
95.01
97.10
2.09
93.12
94.39
+1.27
PR
96.07
97.64
1.57
96.13
97.03
+0.90
-32.15
93.07
96.80
3.73
95.52
97.75
+2.23
30.0
87.52
91.53
94.15
+2.27
ZH
54.15
64.44
+10.29
55.60
64.91
+9.31
+15.4%
Avg
88.51
92.02
3.52
88.69
91.22
+2.53
15.0
Table 9: Comparison of Wikidata (W) and Wikidata + M-NTA (+M-NTA) on entity and name coverage for entity-type queries in MKQA.
0.0
DE
EN
ES
FR
JA
ZH
Avg
Figure 8: Reduction rate in the number of unanswerable queries in MKQA when using M-NTA to improve the coverage of Wikidata.
name for an entity instcad of the complete list of valid names. However, the performance in preci- sion decrease usually decreases, as we hypothesize that there are entities for which it is more difficult to identify incorrect entity names.
the entity in the target language, then knowledge- based QA system will not be able to provide a correct answer. Therefore, increasing the coverage of entity names across languages is essential to ex- tend the support of knowledge-based QA systems to multilingual settings.
E Impact on Downstream Tasks: Question Answering
In section 6, we have investigated the impact of in- creasing coverage and precision of textual informa- tion in two downstream tasks, namely, multilingual entity linking and multilingual knowledge graph completion. Here, we also investigate the impact of our work on Question Answering (QA), with a specific focus on knowledge-seekine queries. One of the main characteristics of knowledge-seekine queries is that they can be usually answered by nav- igating knowledge graph and returning (the name of) entity, e.g., the answer to the query "What is the highest mountain in Washington, US?" is Mount Rainier (Q194057) However, if the knowl- edge graph does not provide a lexicalization for
To quantify the impact of M-NTA on QA, we consider the subset of queries in MKQA (Long- pre et al., 2021), a multilingual QA dataset for Snowledge-seeking queries, whose type of an- swer is classified as "entity", i.e., those queries that can be answered by providing the name of Wikidata entity. Importantly, the original au- thors of MKQA manually added names (primary names and aliases) for all those Wikidata enti- ties that did not have a lexicalization. Therefore. there a set of questions in MKQA which are "unanswerable" by a knowledge- based QA sys- tem that relies on Wikidata; this set of unanswer-

---

able questions impose an upper bound the re- sults achievable by any knowledge-basse QA sys- tem. More specifically, we measure the number answerrble/manswerbbbe queries when relying only on Wikidata' compared to using an M-NTA- augmented Wikidata (Wikidata + M-NTA) in two settings:
Entity coverage: the number of entities in the answers of MKQA for which Wikidata (or Wikidata M-NTA) can provide at least one name;
Name coverage: the number of names for the entities in the answers of MKQA that are also present in Wikidata (or Wikidata M-NTA).
As can see in Table 9, using M-NTA allows us increase the number of answerable queries both when we look entity coverage (+3.52% absolute improvement) and name coverage (+2.53% abso- lute improvement). Notably, M-NTA provides significant increase in entity coverage for simplified Chinese (+ 10.29% absolute improvement), which is the language with lowest coverage, but also in En- glish (+0.14% absolute improvement). Although the absolute improvement in English seems small, entity coverage in English is already high in Wiki- data (99.09%): another way look this im- provement is by analyzing the reduction rate the number of unanswerable queries. As we can see in Figure 8, the reduction rate in the number of unanswerable queries in MKQA can be reduced significantly when using M-NTA to improve the coverage of Wikidata. Even for English, the re- duction rate is about 15.4%, which becomes as high as 52.0% and 53.8% in German and Italian, respectively.
"As of April 2023.


Reviews:
[{"review_id": "B3BtJMfuMr", "reviewer": "Reviewer_sJ2r", "summary": "", "questions": "Just double checking the interpretations I've made\n\n- 324-330 - this specifies that the 1000 entities are likely different (barring some overlap) across languages. Is this the case? Or are they the same 1000 entities across all languages? \n\n- Table 2: For a given column say AR, is it fair to think the F1 score here for coverage implies that for each of the 1000 entities in the WikiKGE-10 for AR, their corresponding entities in English (or other languages in case of MT) are used as a starting point to translate them back to AR and then compare it with the ground truth?\n\n(some clarification here can help make this more concrete)", "limitations": "", "rating": 4, "confidence": 4, "soundness": 5, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your review! We are truly grateful that our contributions — our analysis, our manually-created multilingual benchmark, our proposed methodology, and its usefulness in downstream applications — have been recognized to such an extent. We hope that this response can further solidify your understanding of our work.\n\n__Questions__\n\nYour interpretations are correct! We will use the extra page in the camera ready to provide additional details. In the meanwhile, we hope you can find the following answers helpful.\n\n* __“the 1000 entities are likely different (barring some overlap) across languages”:__ Yes, the 1000 entities in WikiKGE-10 (our manually-curated benchmark) are different across languages.\n    * __Additional details.__ The difference between languages comes from two main factors: First, each sample of 1000 entities is drawn independently for each language. Second -- and most interesting in our opinion -- the entities in the top-10% are different in each language *l*, as we determine entity popularity according to the page views of an entity in *l*. Therefore, the composition of the entities in our benchmark is also affected by culture-specific aspects.\n\n* __“their corresponding entities in English (or other languages in case of MT) are used as a starting point”:__ Yes, the starting point is the existing Wikidata information in a source language. This information is “translated” using MT, LLM prompting, WS, or M-NTA into a target language. If the information of an entity *e* is correctly translated to the target language, then we say that *e* has been “covered”.\n    * __Additional details.__ Notice that a source language (even English) may have no information about some entities. In this case, MT/WS/LLMs will be unable to “translate” the information in any target language. This is exactly one of the strengths of our proposed approach (M-NTA), which can draw information from multiple languages instead of a single one. One could argue that LLMs can be prompted with information coming from multiple source languages too; we leave this investigation to future work, as it goes beyond a “resources and evaluation” paper, while noting that M-NTA can potentially improve further if we improve the underlying LLM.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- The authors recognize and empirically show disparity in knowledge graphs across languages. The exploratory analysis done here is solid featuring a diverse, reasonably sized sample set of languages.\n- The authors creation of the WikiKGE-10 dataset is a valuable resource that is reasonably sized covering 10 diverse languages. The annotation process is described in good detail making it a strong, objective benchmark to evaluate systems against.\n- Their proposed method of combining multiple processes alongside experimental valuation is thorough and indicates strong improvements. The accompanying analyses and insights are also valuable on their own to understand broader behavior of machine translation, web search & llms in the context of cross-lingual knowledge transfer\n- Their downstream evaluation is also thorough corroborating their broader claim that enhanced multilingual knowledge graphs are useful\n- Clearly highlighted limitations of their dataset and methodology\n- Finally, a very comprehensive, detailed appendix that complements the main paper\n\nOverall, this is a very well written paper that is very clear and focused in claims made and is corroborated with rigorous experimental results.", "reasons_to_reject": "Nothing major that I can think of.", "questions_to_authors": "Just double checking the interpretations I've made\n\n- 324-330 - this specifies that the 1000 entities are likely different (barring some overlap) across languages. Is this the case? Or are they the same 1000 entities across all languages? \n\n- Table 2: For a given column say AR, is it fair to think the F1 score here for coverage implies that for each of the 1000 entities in the WikiKGE-10 for AR, their corresponding entities in English (or other languages in case of MT) are used as a starting point to translate them back to AR and then compare it with the ground truth?\n\n(some clarification here can help make this more concrete)", "paper_topic_and_main_contributions": "Problem: Knowledge graphs exhibit significant disparities in quality and data sparsity when dealing with non-English languages.\n- The authors first explore this phenomenon for Wikidata and report lower coverage numbers for 9 different languages when compared to English\n- The authors then create an annotated dataset called WikiKGE-10 that encompass 1000 entities listing their mentions (missing and otherwise) along with errors\n- To improve multilingual coverage & accuracy for knowledge graphs (they call this Knowledge Graph Enhancement/KGE), the authors propose a method they term M-NTA as an ensemble of methods from machine translation, web search & language models\n- They evaluate each of the above methods independently and compare them with that of their proposed M-NTA method for the WikiKGE-10 dataset\n- Finally, they evaluate the value of an enhanced Knowledge Graph for Knowledge Graph Completion & Question Answering\n\nAlong the way, the authors clearly express their motivations backed by exploratory analysis and follow it up with a proposed solution backed by thorough experimental evidence. ", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Typos_Grammar_Style_and_Presentation_Improvements": "- 008 - typo - comparatively scarce*\n- 211 - typo - their content*"}, "extra_text": {}}, {"review_id": "2p0xhgb3UR", "reviewer": "Reviewer_JXeC", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your positive review: we really appreciate that you also believe *“covering non-English languages is always a plus”*!\n\nWe will make sure to include the version of ChatGPT (which we used between March and May) in the camera ready of the paper.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The paper is well-written and motivated and represents a contribution to the field.\nThe aim of covering non-English languages is always a plus.\n", "reasons_to_reject": "I do not see any reason for not accepting the paper.", "questions_to_authors": "", "paper_topic_and_main_contributions": "Authors propose a methodology for knowledge graph enhancement (KGE), intended as the task of increasing coverage and precision of textual information, specifically for non-English languages and information.\nWikiKGE-10, a resource for benchmarking data-centric-AI approaches on KGE of entity names in 10 languages, is presented.\nStarting from the limitations of three well-known approaches dealing with KGE (machine translation, Web search, and large language models), authors propose M-NTA: Multi-source Naturalization, Translation, and Alignment, an approach which exploits all the three aforementioned methods.\n\nIt would be better specifying the version of ChatGPT used in the experiment (even better specifying the period in which the tests have been conducted).\n", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Typos_Grammar_Style_and_Presentation_Improvements": "Line 078 - The acronym AI should be introduced"}, "extra_text": {}}, {"review_id": "QXnDqgwFgi", "reviewer": "Reviewer_N39A", "summary": "", "questions": "- A - What platform did you use for collecting the annotations? How did you compute the pairwise inter-annotator agreement values you report in Figure 2 of Appendix A2? Is it the average of all pairwise comparisons? How many annotators were involved in the annotation process and what is the average number of annotations they did? Line 333 states that besides rating the multilingual entity name/labels, the human graders also added new valid names. How were these added names validated? Were they put into the annotation pool for other annotators to verify too? Does the line \"correct not in Wikidata\" in Table 1 refer to these added names? \n- B - How exactly do you compute the coverage and precision in Table 2? i.e. The increase in coverage by M-NTA for each language is measured with respect to what? The paper will also benefit from at least a few lines clearly stating how this evaluation was done (e.g. how the train test splits were done, random split of names or random split of entities etc). \n- C - Section 4.1. Your method seems to rely on Wikidata text descriptions (Line 440), which you also state are significantly missing in Wikidata, can other types of text be used instead?", "limitations": "", "rating": 3, "confidence": 3, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We really appreciate the time that went into writing this review; we hope this answer can clarify our work and address your concerns about the soundness of our work!\n\n__Questions For The Authors__\n\n* __A - questions about the annotation process__\n    * __“What platform did you use for collecting the annotations?”__ We used an internal annotation tool with an interface that we designed specifically for this task. In case of acceptance, we will expand Appendix A with screenshots that show the same UI that the annotators used. Using our own annotation platform allows us to have better control over the annotators, including training them on the task using a separate dataset, checking their language qualifications, possibility to directly reach out to an annotator, custom entrance tests, regular quality checks, etc.\n    * __“How did you compute the pairwise inter-annotator agreement values you report in Figure 2 of Appendix A2?”__ We report pairwise inter-annotator agreement using (an extension of) Cohen’s Kappa: each item was annotated by 3 annotators, and we computed the agreement by considering all rating pairs. Although Cohen’s Kappa is usually used to compute IAA between 2 raters, its extension to $n > 2$ raters, known as Fleiss’ Kappa, can sometimes lead to paradoxical results (*“Handbook of Interrater Reliability”*, Kilem Gwet) and it’s not widely adopted by the NLP community. Instead, we also report IAA using Krippendorff’s Alpha (Figure 3), which can be applied to $n > 2$ raters and also takes into account the cardinality of the ratings.\n    * __“How many annotators were involved in the annotation process and what is the average number of annotations they did?”__ The number of annotators varied from language to language. On average, we employed 21 annotators per language; each rater annotated around 420 items (+50 for training). \n    * __“How were these added names validated? Were they put into the annotation pool for other annotators to verify too?”__ Yes, new names were re-validated by 3 annotators using the same process.\n        \n* __B - questions about the evaluation__\n    * __“How exactly do you compute the coverage and precision in Table 2?”__ We will expand our description of coverage and precision with a formal definition in the extra page of the camera ready in case of acceptance. In our case, we say that a method “covers” an entity $e$ in a language $l$, if it is able to provide a valid entity name for $e$ in $l$ (exact string matching with lower casing and punctuation stripping). The coverage score indicates the percentage of entities covered by a given method in $l$. Similarly, the precision score indicates the percentage of cases in which a given method can identify that an entity name in Wikidata is actually incorrect (i.e., a name that is considered incorrect by our annotators in WikiKGE-10).\n    * __“how the train test splits were done, random split of names or random split of entities?”__ We note that all the methods we investigate (even our method M-NTA) are unsupervised, that is, they are not trained on a task-specific training set. For what concerns our test set, we mention the sampling process in Section 3.4: we created our test set (WikiKGE-10) by randomly sampling 1000 entities from the top-10% of the popular entities in Wikidata, sorted by the number of page views in each language of interest. \n        \n* __C - questions about the methodology__\n    * __“[naturalization in M-NTA] can other types of text be used instead?”__ Yes! As M-NTA already yields positive results, we left the investigation of more complex methods to future work. However, the naturalization step of M-NTA can be immediately improved: for example, one could use graph-to-text approaches for cases in which an entity description is not available.\n\n\n__Typos, Grammar Style And Presentation Improvements__\n\nWe would also like to address the following points (“the claims made in parts of the the abstract, introduction and conclusion do not fully match with the experimental work carried out”), hoping to clarify the soundness of our work.\n\n* __“the term \"textual description\" throughout [the paper], which can be read as including the \"entity description\" task\"__ We use the term \"textual description\" only once in the paper and with the same meaning as \"entity description\". If the Reviewer refers to \"textual information\", we use this term to refer to language-dependent \"string\" information in a KG. Although our paper does not include an intrinsic evaluation of entity descriptions, we still describe how our method can be applied to them (Appendix B.7), provide a qualitative analysis with some output examples (Table 8), and — most importantly — include an extrinsic evaluation of their downstream impact (see next answer).\n* __“[the paper] does not contain any data or experimental tasks that investigates this task [entity descriptions]”__ Our paper does contain experiments that involve entity descriptions. In our experiments on MKGC (Section 5), the model also uses entity descriptions to create entity representations (line 610). It is interesting to notice that the dataset used for MKGC is constructed to have complete coverage of entity names; this suggests the improvements come from better coverage (and quality) of the entity descriptions rather than the names. We will expand the camera ready with this information and examples, hoping to make it clearer that our experiments extrinsically evaluate the impact of better entity descriptions.\n* __“the experiments and results in the following sections are only on multilingual entity name[s]”__ Please, see answer above.\n* __“maybe this may be better characterised as a method for data improvement rather than the actual task of question answering?”__ Indeed, our contributions fall into the category of data-centric AI approaches (lines 77—81; 272-280; 315-317; 627-633). In fact, our work is a way to improve the data (KG), and we show that our proposed methodology for improving textual data in a KG is a simple and effective way for better downstream results in three tasks without modifying the models.\n* __“What is the specific risk [of using LLMs], and are you able to point to the parts of your experimental results that supports this claim?”__ When comparing the mistakes of two systems, e.g., LLMs and web search, a metric may hide the relative gravity of the two mistakes, and we believe that it is important for our readers to be aware of this. More specifically, our qualitative analysis suggests that a mistake made by web search (which usually returns semantically similar results) may be less problematic than a mistake made by an LLM (which can generate completely unrelated and erroneous results more easily). Therefore, depending on the downstream application of interest, we argue that one may still prefer web search to BLOOMZ even if the former slightly underperforms the latter in terms of numerical results in our benchmark to avoid the cases shown in Section 4. Finally, we note that we did not limit our analysis to the entities in the top-10%, hence, we also mention long-tail entities.\n\n__References__\n\nThank you for pointing out those papers! We will extend the Related Work to include them and describe the high-level differences with our work, especially between JRC-Names and WikiKGE-10.\n\nIn short: i) WikiKGE-10 is completely manual; ii) WikiKGE-10 is mapped 1-to-1 to Wikidata; iii) WikiKGE-10 is not limited to persons and organizations; iv) JRC-Names considers names with spelling mistakes as valid names (as they may appear in real-life scenarios), whereas WikiKGE-10 considers them incorrect (as we aim for a clean KG); v) JRC-Names does not distinguish between entities that have the same name (“very likely that different persons sharing the same first and last name have the same identifier because no disambiguation mechanism is in place”).", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The human-verified evaluation dataset collected by the authors can be useful to supplement another multilingual entity name/label dataset that was released in 2011 (Steinberger et al, 2017) by introducing new and more recent entities for evaluation.  An automatic method to help plug the many missing multilingual entity names in knowledge bases can be helpful, especially for downstream knowledge base QA models to use. More immediately, it can also allow wider access to/understanding of such knowledge graph information by non-English speakers.", "reasons_to_reject": "The claims made in parts of the the abstract, introduction and conclusion do not fully match with the experimental work carried out (see points 1,2,4 in **Typos Grammar Style And Presentation Improvements** below). There are questions about the procedures for obtaining the paper's experimental results. For example, it is not clear how the coverage and precision scores in Table 2 were computed.", "questions_to_authors": "- A - What platform did you use for collecting the annotations? How did you compute the pairwise inter-annotator agreement values you report in Figure 2 of Appendix A2? Is it the average of all pairwise comparisons? How many annotators were involved in the annotation process and what is the average number of annotations they did? Line 333 states that besides rating the multilingual entity name/labels, the human graders also added new valid names. How were these added names validated? Were they put into the annotation pool for other annotators to verify too? Does the line \"correct not in Wikidata\" in Table 1 refer to these added names? \n- B - How exactly do you compute the coverage and precision in Table 2? i.e. The increase in coverage by M-NTA for each language is measured with respect to what? The paper will also benefit from at least a few lines clearly stating how this evaluation was done (e.g. how the train test splits were done, random split of names or random split of entities etc). \n- C - Section 4.1. Your method seems to rely on Wikidata text descriptions (Line 440), which you also state are significantly missing in Wikidata, can other types of text be used instead?", "paper_topic_and_main_contributions": "This work is on the task of obtaining multilingual names/labels for entities present in a knowledge graph, which can be useful for plugging such missing information in large knowledge graphs (that are typically only well-populated for languages where Wikipedia articles and editors number the most, e.g. English). The authors collected a human-verified dataset of approximately 35,000 multilingual names for 10 languages across different typologies and used this for their evaluation. The authors propose a method for the task by combining three means to obtain these multilingual names (machine translation, web search and prompting large language models), and which ensures that only name/label candidates that is supported by another method is accepted. They report increase in coverage and precision (see questions below) from their method, and also carried out experiments with multilingual entity linking and multilingual knowledge graph completion to show that adding predictions from their method improves these downstream tasks.", "excitement": 3, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {"Missing_References": "- Generating Multilingual Descriptions from Linguistically Annotated OWL Ontologies: the NaturalOWL System (Galanis and Androutsopoulos, 2007)\n- Exploiting OWL Ontologies in the Multilingual Generation of Object Descriptions (Androutsopoulos et al 2005)\n- JRC-NAMES: A Freely Available, Highly Multilingual Named Entity Resource (Steinberger et al, 2011)", "Typos_Grammar_Style_and_Presentation_Improvements": "- The task of generating names and entity descriptions have meaningful differences. Compared to multilingual entity name extraction/retrieval/generation (which is the subject of the new benchmark dataset collected and the experimental work in this paper), the latter involves additional NLG subtasks such as content selection (what information to realize in the text) and surface realisation (for grammaticality and fluency). The main body of the paper currently refers heavily to the entity description task, but does not contain any data or experimental tasks that investigates this task. Some suggestions: It will be better for the reader if you clearly refer to the task you investigated (e.g. multilingual entity name translation/retrieval/generation) instead of using the term \"textual description\" throughout, which can be read as including the \"entity description\" task. The \"entity description\" task could then be introduced and discussed on a future work/directions section nearer the conclusion. The paper could also benefit from a title that more clearly describes the actual task you investigate (it is too broad and general currently). \n- Line 365: Better to point the reader to the precise subsection (i.e. appendix B7). Also, it does not seem accurate to say \"but we also **show** that the methodologies discussed in this section can be extended to other types of textual information, such as entity descriptions...\", since the experiments and results in the following sections are only on multilingual entity name retrieval/extraction/generation task.  Perhaps you mean \"we also suggest some ways the methodologies...\"\n- Line 549: This paragraph can benefit from more clarity as it appears at odds with other parts of the paper. In Section 3.4 you state that WikiKGE-10 was created by querying top-10% head entities, but in this section you refer to long-tail entities. It is also not clear what you mean by \"the risk of using them is higher in case of error\". What is the specific risk, and are you able to point to the parts of your experimental results that supports this claim?\n- Line 027, 104 and 547: Although you include (in the appendix) a study about how using M-NTA can reduce the number of questions in the MKQA dataset that require human manual annotation to obtain multilingual entity names as answers, maybe this may be better characterised as a method for data improvement rather than the actual task of question answering? \n- Line 475: it may be better to state and describe your choice of phi (the similarity measure) here, or at least point the reader to the appendix where you mention that it is exact string match (after lowercasing and stripping punctuation)."}}]

Author rebuttal:

Meta-review: This paper is concerned with knowledge graph enhancement to improve the coverage and precision; it specifically aims to improve entity information in 10 languages, and presents a large new dataset (35,000 multilingual names) as the WikiKGE-10 benchmark. The core idea of the approach is to combine previous approaches to this problem (leveraging machine translation, web search and LLMs) into their approach M-NTA "Multi-source Naturalization, Translation and Alignment". 
The reviewers agree that the paper is well-written and sound; they particularly appreciate the crosslingual work and the evaluation, as well as the paper's clarity regarding limitations. Some more detailed points questions about the methods could be addressed in the rebuttal and should help to improve the final version of the paper.
Decision: Accept-Main