Evaluating the Knowledge Base Completion Potential of GPT
Authors: Blerta Veseli
Simon Razniewski
Jan-Christoph Kalo
Gerhard Weikum
Venue: EMNLP 2023 Findings (2023)
Keywords: knowledge base completion
knowledge graphs
probing language models
evaluation
TL;DR: A realistic assessment of the potential of a large language models (GPT-3, GPT-35, GPT-4) for knowledge base completion.

Abstract: Structured knowledge bases (KBs) are an asset for search engines and other applications but are inevitably incomplete. 
Language models (LMs) have been proposed for unsupervised knowledge base completion (KBC), yet, their ability to do this at scale and with high accuracy remains an open question. 
Prior experimental studies mostly fall short because they only evaluate on popular subjects, or sample already existing facts from KBs. 
In this work, we perform a careful evaluation of GPT's 
potential to complete the largest public KB: Wikidata. 
We find that, despite their 
size and capabilities, 
models like GPT-3, ChatGPT and GPT-4 do
not achieve fully convincing results on this task. 
Nonetheless, it provides solid improvements over earlier approaches with smaller LMs. 
In particular, we show that it is feasible 
to extend Wikidata by 27M facts at 90% precision.

Evaluating the Knowledge Base Completion Potential of GPT
Blerta Veseli', Simon Razniewski", Jan-Christoph Kalo", Gerhard Weikum Max Planck Institute for Informatic'', Bosch Center for AI2, University of Amsterdam" 3 (weikum, jreselij@mppiiiee wtt.p//iif..p..deeee simon. razni wwwaziiewsk@ddeebee.... wwwch.sccc...oo.. com j. kalo@uva.nnn
Abstract
the seminal LAMA paper (Petroni et al., 2019), a throve of works have explored how better probe, train, or fine-tune these LMs (Liu et al., 2022).
Structured knowledge bases (KBs) are an asset for search engines and other applications, but are inevitably incomplete. Language models (LMs) have been proposed for unsupervised knowledge base completion (KBC), yet, their ability to do this at scale and with high accuracy remains an open question. Prior experimental studies mostly fall short because they only eval- uate on popular subjects, or sample already ex- isting facts from KBs. In this work, we perform a careful evaluation of GPT's potential to com- plete the largest public KB: Wikidata. We find that, despite their size and capabilities, mod- els like GPT-3, ChatGPT and GPT-4 do not achieve fully convincing results on this task. Nonetheless, they provide solid improvements over earlier approaches with smaller LMs. In particular, we show that, with proper threshold- ing, GPT-3 enables to extend Wikidata by 27M facts at 90% precision.
Nonetheless, we observe a certain divide be- tween these late-breaking investigations, and prac- tical KB completion. While recent LM-based ap- proaches often focus on simple methodologies that produce fast results, practical KBC so far is highly precision-oriented, extremely laborious process, in- volving very high degree of manual labour, ei- ther for manually creating statements (Vrandečić and Krötzsch, 2014), or for building comprehen- sive scraping, cleaning, validation, and normaliza- tion pipelines (Auer et al., 2007; Suchanek et al., 2007). For example, part of Yago's success stems from its validated >95% accuracy, and according to (Weikum et al., 2021), the Google Knowledge Vault was not deployed into production partly be- cause it did not achieve 99% accuracy. Yet, many previous LM analyses balance precision and recall or report precision/hits @k values, implicitly tuning systems towards balanced recall scores resulting in impractical precision. is also important to keep in mind the scale of KBs: Wikidata currently contains around 100 million entities and 1 1.2B statements.
1 Introduction
Structured knowledge bases (KBs) like Wiki- data (Vrandečié and Krötzsch, 2014), DBpe- dia (Auer et al., 2007), and Yago (Suchanek et al., 2007) are employed in many knowledge-centric applications like search, question answering and dialogue. Constructing and completing these KBs at high quality and scale is a oong-standing re- search challenge, and multiple benchmarks exist, e.g., FB15k (Bordes al., 2013), CoDEx (Safavi and Koutra, 2020), and LM-KBC22 (Singhania et al., 2022). Text-extraction, knowledge graph embeddings, and LM-based knowledge extraction have continuously moved scores upwards on these tasks, and leaderboard portals like Paperswitheode! provide evidence for that.
The cost of producing such KBs is massive. An esti- mate from 2018 sets the cost per statement at for manually curated statement, and ct for automati- cally extracted ones (Paulheim, 2018). Thus, even small additions in relative terms might correspond to massive gains in absolute numbers. For exam- ple, even by the lower estimate of et/statement, adding one statement to just 1 1% of Wikidata hu- mans would come at cost of 100,000 $.
In this paper, we conduct systematic analysis of the KB completion potential of GPT, where we focus high precision. We evaluate by employ- ing (i) recent KB completion benchmark, WD- KNOWN, (Veseli et al., 2023), which randomly samples facts from Wikidata and (ii) by manual evaluation of subject-relation pairs without object
Recently, LMs have been purported promis- ing source of structured knowledge. Starting from
hhtpps://papeee swi wwwp task/ knowledge- grwph-capoeooo completion

---

values. Our main results are:
LMs for KB construction Knowledge extrac- tion from LMs provides fresh hope for the syn- ergy of automated approaches and high-precision curated KBs. It provides remarkably straightfor- ward access to very large text corpora: The ba- sic idea by (Petroni et al., 2019) is to just de- fine one template per relation, then query the LM with subject-instantiate versions, and retain its top prediction(s). A range of follow-up works ap- peared, focusing, e.g., on investigating entities, im- proving updates, exploring storage limits, incorpo- rating unique entity identifiers, and others (Shin et al., 2020; Poerner et al., 2020; Cao et al., 2021; Roberts et al., 2020; Heinzerling and Inui, 2021;
1. For the long-tail entities of WD-KNOWN, GPT models perform considerably worse than what less demanding benchmarks like LAMA (Petroni et al., 2019) have indicated. Nonethe- less, we can achieve solid results for language- related, socio-demographic relations (e.g., na- tiveLanguage).
2. Despite their fame and size, out of the box, the GPT models, including GPT-4, do not pro- duce statements of a high enough accuracy as typically required for KB completion.
Petroni et al., 2020; Elazar et al., 2021; Razniewski et al., 2021: Cohen et al., 2023; Sun et al.. 2023). Nonetheless, we observe the same gaps above: The high-precision area, and completion of already existing resources, are not well investigated.
3. With simple thresholding, for the first time, we obtain method that can extend the Wikidata KB at extremely high quality (>90% preci- sion), at the scale of millions of statements. Based on our analysis of 41 common relations, we would be able to add a total of 27M high- accuracy statements.
Several works have analyzed the potential of larger LMs, specifically GPT-3 and GPT-4,. They investigate few-shot prompting for extracting fac- tual knowledge for KBC (Alivanistos et al., 2023) or for making the factual knowledge in a LM more explicit (Cohen et al., 2023). These models can aid in building a knowledge base on Wikidata or improving the interpretability of LMs. Despite the variance in the precision of extracted facts from GPT-3, it can peak at over 90% for some relations. Recently, GPT-4's capabilities for KBC and rea- soning were examined (Zhu et al., 2023). This research compared GPT-3, ChatGPT, and GPT-4 on information extraction tasks, KBC, and KG- based question answering. However, these studies focus on popular statements from existing KBs, neglecting the challenge of introducing genuinely new knowledge in the long tail.
2 Background and Related Work
KB construction KB construction has consid- erable history. One prominent approach is by hu- man curation, as done e.g., in the seminal Cyc project (Lenat, 1995), and this is also the back- bone of today's 'sssi most prominent public KB, Wiki- data (Vrandečić and Krötzsch, 2014). Another popular paradigm is the extraction from semi- structured resources, as pursued in Yago and DBpe- dia (Suchanek et al., 2007; Auer et al., 2007). Ex- traction from free text has also been explored (e.g.. NELL (Carlson al., 2010)). A popular paradigm link prediction, e.g.,
has been embedding-basee via tensor factorization like Rescal (Nickel al., 2011), 1). and KG embeddings like TransE (Bordes et al., 2013).
In (Veseli al., 2023), we analyzed to which degree BERT can complete the Wikidata KB, i.e. provide novel statements. Together with the focus on high precision, this is also the main difference of the present work to the works cited above, which evaluate on knowledge already existing in the KB, and do not estimate how much they could add.
An inherent design decision in KBC is the P/R trade-off academic projects often open to trade these freely (c.g., F-I
F-1 scores), yet produc- tion environments are often critically concerned with precision, c.g., Wikidata generally discour- aging statistical inferences, and industrial players likely use to a considerable degree human editing and verification (Weikum et al., 2021).
3 Analysis Method
Dataset We consider the 41 relations from the LAMA paper (Petroni et al., 2019). For automated evaluation and threshold finding, we employ the -N-KNNW dataset (Veseli et al., 2023). Unlike other KBC datasets, this one contains truly long-
For example in all of Rescal, TransE, and LAMA, the main results focus on metries like hits @k MRR, or AUC, which provide no bounds on precision.

---

GPT-3
CIoRGPT
sired value. We threshold only GPT-3, because only GPT-3's token probabilities are directly ac-
GPT4
text davinci 003
gpt.3.5 turbo
turbo
Relation
F1
P
R
F1
F1
writtenn
0.62
0.59
0.6
0.91
0.78
0.84
0.37
0.85
0.52
cessable in the API, and because the chat-aligned models do not outperform it in the retain-all all setting.
ownedity
0.41
0.3
0.44
0.51
8:17
0.27
angivoLL
0.85
0.85
0.85
0.69
0.86
0.77
0.53
0.88
0.66
LanguageOfFilm
0.78
0.7
0.52
Approaches to estimate probabilities post-hoe can be found in (Xiong et al., 2023).
0.77
0.48
0.59
0.77
0:44
0.56
0.49
0.86
0.62
othcialLanguage
0.62
0.6
0.64
0.65
0.39
foundedia
0.27
0.53
0.36
0.4
0.38
0.39
0.14
0.58
0.23
play Instrument
0.25
0.36
0.3
0.16
0.18
0.17
0.13
0.6
0.21
Since automated evaluations are only possi- ble for statements already in the KB, in a sec- ond step, we let human annotators evaluate the correctness of 800 samples of novel (out-of-KB) High-aceuracy predictions. We hereby use a relation-specific threshold determined from the au- tomated 75%-9595 95% precision range. MTurk amno-
pan(Or
8:05
0.1
0.17
0.13
0.06
0.1
citizenOr
0.72
0,62
0.67
0.67
0.6
0.63
0.47
0.58
0.56
spokeal anguago
0.76
MayerPosition
0.4
0.4
04
0.18
0.24
0.21
0.23
0.7
0.35
Continent
0.4
namedAfier
0.5
0.49
0.49
0.53
0.44
0.48
0.12
0.36
0.18
CosmCoy
0.77
0.5
0.61
0.75
0:48
0.59
0.4
0.55
0.46
musacl.be Jabel
18.99
0.51
0.3
0.16
0.18
0.17
D.08
0.88
0.14
hasReligiion
0.44
0.36
0.4
0.47
0.38
0:42
0.16
0.39
0.25
developediyy
0.43
0.47
0.6
0.19
Ouunisittionn
0.38
0.24
0.29
0.52
0.22
0.4
0.42
0.33
0.37
subelaso(If
00:11
0,4
tators could use Web search to verify the correct- 0.36
diplonaticRelation
0.36
0.62
0.16
0.7
0.17
0.27
0:11
0.32
ness of our predictions on a 5-point Likert scale (correct/ wttpllllliliiooo///aaaasallleeee counted predictions that were rated correct likely as true predictions, and all others as false.
OfuriiyOOrrigin
0.6
0.34
0.43
0.48
0.31
0.38
0.2
0.39
0.26
We
Macro-Aserag
0.49
0.48
0.47
0.53
0.46
0.48
10.28
0.59
0.36
Table 1: Automated evaluation in the retain-all setting: GPT-3 (text-davinci-003 with 175B parameters), GPT-4 (#parameters unknown) and ChatGPT (gpt-3.5-turbl with #parameters unknown) on 1000 samples/relation from WD-Known
Prompting setup To query the GPT models, we utilize instruction-free prompts listed in the appendix. Specifically for GPT-3, we follow the prompt setup of (Cohen et al., 2023), which is based on an instruction-fis prompt entirely consisting of 8 randomly sampled and manually checked examples. In the default setting, all exam- ple subjects have at least one object. Since none of the GPT models achieved precision >90% and we can only threshold GPT-3 for high precision, we focus on the largest GPT-3 model (text-davinci- 003) in the following. We experimented with three variations for prompting this model:
tail entitics, by randomly sampling from Wikidata, a total of 4 million statements for 3 million sub- jects in 41 relations (Petroni et al., 2019). Besides this dataset for automated evaluation, for the main results, we use manual evaluation on Wikidata enti- ties that do not yet have the relations of interest. For this purpose, for each relation, we manually define a set of relevant subject types (e.g.. soat for developedBy), that allows us to query for subjects that miss a property.
Following (Cohen et al., 2023), in this variant, we manually se- lected 50% few-shot examples, where GPT-3 did not know the correct answer, to teach the model to output "Don't know". This is sup- posed to make the model more conservative in cases of uncertainty.
Examples w/o answer:
Evaluation protocol In the automated setting, we first use retain-all setting, where we evalu- ate the most prominent GPT models (GPT-3 text- davinci-003, GPT-4, and ChatGPT gpt-3.5-turbo) by precision, recall, and Fl. Table 1 shows that none of the GPT models could achieve precision of >90%. In a second step, the precision- setting, we therefore sort predictions by confidence and evaluate by recall at precision 95% and 90% (R @ P95 and @ P90). To do so, we sort the pre- dictions for all subjects in a relation by the model's probability on the first generated token?, then com- pute the precision at each point of this list, and return the maximal fraction of the list covered while maintaining precision greater than the de-
2. Textual context augmentation: Following (Petroni et al., 2020), we test whether adding textual context improves model performance. We hereby employ Google Web Search, with the subject and relation of interest as search query. The top 1 result snippet is then in- cluded as context to the prompt.
3. #few-shot examples: A standard parameter in prompting is the number of few-shot exam- ples. They have huge impact on monetary costs. We vary this number between I and 12.
This is a heuristic only, as unbiased probabilities are not easy to assign to multi-token generations in list answers (Sing- hania ct al., 2023).

---

Acurrent stris. Wikidata
Bsubj. missing simbs.
Traction for which GPT-3 can
manual
relative gnowth
Relation
give hight confieeodddee prediction
#addable simts.
foundedin
43,254
85%
43%
citizenOE
4,206,684
4,616,001
536
230,830
82%
5%
country Olturisiicionn
901,0060
24,906
18.974
namedAfter
340,231
477.545
22%
105. 125
611%
20%
inCominent
71.101
ownddRy
449,140
416,457
655
24,986
24%5
1%
houtCuuity
14,275,596
18,663
spokenLanggugg
2,148,775
7,134,533
57%
4,066.68 689
92%
174%
writtentn
14,140,455
24,9900161
73%
18,242,877
119%
officiall anguage
19,678
6,776
4256
2,846
1001%
14%
developedlBy
42,379
29.349
6%
1.761
94%
OfOrign
1,296,038
135,196
40%
66.246
3015
2%
hasCapital
111,171
973
11%
107
1.1%
0%
LanguageO/Film
337,682
70,069
2156
16,961
82%
4%
nativelLanguuge
264,778
7,871,085
19%
3,856.831
82%
1195%)
shuresRoddors
6,946
7255
Overall
38,684,975
46,934,749
66%
27,224,452
90.5
70%
Table 2: Manual evaluation: Wikidata KB completion potential GPT-3 text-davinci-003 with precision-orientee thresholding.
4 Results and Discussion
correlations (person names often give away their nationality), otherwise the task remains hard.
Can GPT models complete Wikidata at preci- sion AND scale? In Table we already showed that without thresholding, none of the GPT models achieve sufficient precision. Table shows main results when using precision-oriented thresh- olding, on the 16 best-performing relations. The fourth column shows the percentage of subjects for which we obtained nigh-confidence predictions, the fifth how these translates into absolute statement numbers, and the sixth shows the percentages that were manually verified as correct (sampled). In the last column, show how this number relates to the current size of the relation.
In Table 3 we report the automated evaluation of precision-oriented thresholding. We find that on many relations, GPT-3 can reproduce existing statements at over 95% precision, and there are sig- nificant gains over the smaller BERT-large model. At the same time, it should be noted that (Sun al., 2023) observed that for large enough models, pa- rameter scaling does not improve performance fur- ther, so is well possible that these scores represent a ceiling w.r.t. model size.
Is this cost-effective? Previous works have esti- mated the cost of KB statement construction at 1 ct. (highly automated infobox scraping) to S2 (manual curation) (Paulheim, 2018). Based our prompt size (avg. 174 tokens), the cost of one query is about 0.35 ct., with filtering increasing the cost per retained statement to about 0.7 et. So LM prompt- ing is monetarily competitive to previous infobox scraping works, though with much higher recall potential.
We find that manual precision surpasses 90% for 5 relations, and 80% for 11. Notably, the best- performing relations are mostly related to socio- demographic properties (languages, citizenship). In absolute terms, we find a massive number of high-accuracy statements that could be added to the writtenin relation (18M), followed by spo- kenLanguage and nativeLanguage (4M each). In
relative terms, the additions could increase the ex- isting relations by up 1200%, though there is surprising divergence (4 relations over 100%, 11 relations below 20%).
In absolute terms, prompting GPT-3 for all 48M incomplete subject-relation pains reported in Table 2 would amount to an expense of $168,000 and yield approximately 27M novel statements.
Does GPT provide quantum leap? Generat-
ing millions of novel figh-precision facts is sig- nificant achievement, though the manually verified precision is still below what industrial KBs aim for. The wide variance in relative gains also shows that GPT only shines in selected areas. In line with pre- vious results (Veseli et al., 2023), we find that GPT can do well on relations that exhibit high surface
Does "Don't 't know" prompting help? In Table 4 (middle) we show the impact of using examples without an answer. The result is unsystematic, with notable gains in several relations, but some losses in others. Further research on calibrating model confidences seems important (Jiang et al., 2021; Singhania et al., 2023).

---

GPT:3
GPT-3
5 Conclusion
1ost-davinci-003
kodtsouri-000
BERT Large
Large
Relation
790.99
700795
700790
700790
We provided the first analysis of the real KB com- pletion potential of GPT. Our findings indicate that GPT-3 could add novel knowledge to Wikidata, at unprecedented scale and quality (27M state- ments at 90% precision). Compared with other approaches the estimated cost of $168,000 is sur- prisingly cheap, and well within the reach of indus- trial players. We also find that, the preeipreciioonn bracket, GPT-3 distills web content to a degree that context augmentation does not easily help.
wrtteedn
0.69
0.76
0.2
0.39
ownalRy
0.37
0.39
0.11
0.16
onzivelinguuge
0.7
0.6
0.21
0.35
0.011
0.11
Castapial
0.19
0.31
0.02
0.02
Languae
Rundedla
0.06
0.08
playsinstrument
0.02
0.02
pantor
0.01
0.01
0.24
0.02
0.03
spokeal.onguuge
0.01
0.47
Q,13
0.24
playerosdtion
0.01
0.02
int Cnminent
0.01
0.01
0
ammllAterr
0.11
huoCountry
0.01
0.34
0.02
0.02
MusicLabel
0:01
0.02
husReligio
developeaRyy
0:11
0.04
0.44
Open issues remain in particular around identi- fying high-confidence predictions within an LM's generations (Jiang et al., 2021; Singhania et al., 2023; Xiong et al., 2023), and the choice of exam- ples.
country Othuradictio
0.08
Q
0
0.02
Q.2
subllesOOr
diplomaticehhation
0.01
CouneryCttiigin
0.01
0.01
0.11
Table 3: Automated evaluation the precision- thresholding setting: GPT-3 (text-davinci-003 with 175B parameters, text-curie-001 with 6.7B parame- ters) and BERT-large (340M parameters) on 1000 sam- ples/relation from WD-Known.
Limitations
Using LMs for automated knowledge generation comes with the standard risk of exacerbating de- mographic biases. For example, many of the best- performing relations are language-related, where the model presumably often estimates a person's 's native language entirely from their name. In terms of reproducibility, it should be noted that our results are tied to closed-source commer- cial API. Although GPT-3/4/chatGPT are widely used in research and industry, and OpenAl has an- nounced plans to keep stable model versions online, long-term -erg-termmoe reproducibility is not ensured, and the internal workings of GPT are publicly not known. Although statement generation is at the core of KB completion, for complete KBC pipeline, we are still missing critical components. This concerns in particular entity disambiguation, which essen- tial for relations with more than a few hundred possible object values. Similarly, Wikidata and other KBs give critical importance to scrutable ref- erencing of statements. This is not easily possible with LMs.
Stundard and texuual context
Standard
Don't know
Relation
RS
R8PP90
RS0P95
R9.P90
RaP
sativeLLmguuge
0.22
0.7
0,56
0.68
0
foundeciln
0.06
0.08
0.09
0.13
0
0.01
developeiRyy
0.11
0.06
11118
0
anguage
0.01
0.01
0.04
employclBy
0.01
1,011
inComiient
citizent3f
0.01
0.34
0
0.24
0.03
0.04
Table 4: Effect of variations to the standard prompting setting.
Does textual context help? Table 4 (right) shows the results for prompting with context. Surpris- ingly, this consistenly made performance worse, with hardly any recall beyond 90% precision. This is contrary to earlier findings like (Petroni al., 2020) (for BERT) or (Mallen al., 2023) (for QA), who found that context helps, especially in the long tail. Our analysis indicates that, the high- precision bracket, misleading contexts cause more damage (lead to high confidence in incorrect an- swers), than what helpful contexts do good (boost correct answers).
References
Dimitrios Alivanistos, Selene Báez. Santamaría, Michael Cochez, Jan-Christoph Kalo, Emile van Kricken, and Thiviyan Thanapalasingam. 2023. Prompting as probing: Using language models for knowledge base construction.
How many few-shot examples should one use? Few-shot learning for KBC works with remarkably few examples. While our default experiments, fol- lowing (Cohen et al., 2023), used examples, we found actually no substantial difference to smaller example sizes as low as 4.
Sören Auer, Christian Bizer, Georgi Kobilarov, Jens Lehmann, Richard Cyganiak, and Zachary G. Ives. 2007. DBpedia: nucleus for a web of open data. ISWC.

---

Antoine Bordes, Nicolas Usumier, Alberto Garcia- Duran, Jason Weston, and Oksana Yakhnenko. 2013. Translating embeddings for modeling multi- relational data. neurIPS.
Nina Poerner. Ulli Waltinger, and Hinrich Schüitze. 2020. E-BERT: Efficient-yer ceeeit---t---fttfteeee effective entity embeddings for BERT. In Findings of EMNLP.
Simon Razniewski, Andrew Yates, Nora Kassner, and Gerhard Weikum. 2021. Language models as for knowledge bases. DL4KG.
Nicola De Cao, Wilker Aziz, and Ivan Titov. 2021. Editing factual knowledge in language models. In EMNLP.
Adam Roberts, Colin Raffel, and Noam Shazeer. 2020. How much knowledge can you pack into the parame- ters of a language model? EMNLP.
Andrew Carlson, Justin Betteridge, Bryan Kisiel, Burr Settles, Estevam Hruschka, and Tom Mitchell. 2010. Toward an architecture for never-ending lan- guage learning. In AAAI.
Tara Safavi and Danai Koutra. 2020. CoDEx: A com- prehensive knowledge graph completion benchmark. In EMNLP.
Roi Cohen, Mor Geva, Jonathan Berant, and Amir Globerson. 2023. Crawling the internal knowledge- base of language models. Findings of EACI.
Taylor Shin, Yasaman Razeghi, Robert L. Logan IV, Eric Wallace, and Sameer Singh. 2020. AutoPrompt: Eliciting knowledge from language models with tomatically generated prompts. In EMNLP.
Yanai Elazar, Nora Kassner, Shauli Ravfogel, Abhilasha Ravichander, Eduard Hovy, Hinrich Schütze, and Yoav Goldberg. 2021. Measuring and improving consistency in pretrained language models. TACL.
Sneha Singhania, Tuan-Phong Nguyen, and Simon Razniewski. 2022. LM-KBC: Knowledge base con- struction from pre-traie pre-trained language models. CEUR.
Benjamin Heinzerling and Kentaro Inui. 2021. Lan- guage models knowledge bases: On entity repre- sentations, storage capacity, and paraphrased queries. In FACI.
Sneha Singhania, Simon Razniewski, and Gerhard Weikum. 2023. Extracting multi valued relations from language models. Repl4NLP.
Zhengbao Jiang, Araki, Haibo Ding, and Graham Neubig. 2021. How can we know when language models know? on the calibration of language models for question answering. TACL.
Fabian M Suchanek, Gjergji Kasneci, and Gerhard Weikum. 2007. Yago: core of semantic knowledge. In wwW.
Douglas B Lenat. 1995. Cye: A large-scale investment knowledge infrastructure. CACM.
Kai Sun, Yifan Ethan Xu, Hanwen Zha, Yue Liu, and Xin Luna Dong. 2023. Hcad-to-tail: How knowl- edgeable are large language models (Il))? aka will 1lms replace knowledge graphs? arXiv.
Pengfei Liu, Weizhe Yuan, Jinlan Fu, Zhengbao Jiang, Hiroaki Hayashi, and Graham Neubig. 2022. Pre- train, prompt, and predict: A systematic survey of prompting methods in natural language processing. ACM CSUR.
Blerta Veseli, Sneha Singhania. Simon Razniewski, and Gerhard Weikum. 2023. Evaluating language models knowledge base completion. In ESWC.
Alex Mallen, Akari Asai, Victor Zhong, Rajarshi Das, Daniel Khashabi, and Hannaneh Hajishirzi. 2023. When not to trust language models: Investigating effectiveness of parametric and non -araamaettic mem- ories. In ACI.
D. Vrandečić and M. Krötzsch. 2014. Wikidata: A free collaborative knowledge base. CACM.
Maximilian Nickel, Volker Tresp, and Hans-Peter Kriegel. 2011. A three-way model for collective learning on multi-relational data. JCMI.
Gerhard Weikum, Luna Dong, Simon Razniewski, and Fabian M. Suchanek. 2021. Machine Knowledge: Creation and Curation of Comprehensive Knowledge Bases. In Fnī.
Heiko Paulheim. 2018. How much is uriple? estimat- ing the cost knowledge graph creation. ISWC.
Miao Xiong, Zhiyuan Hu, Xinyang Lu, Yifei Li, Jie Fu, Junxian He, and Bryan Hooi. 2023. Can Iims express their uncertainty? an empirical evaluation of confidence elicitation in 1lms. arXiv.
Fabio Petroni, Patrick Lewis. Aleksandra Piktus, Tim Rocktäschel, Yuxiang Wu, Alexander H Miller, and Sebastian Riedel. 2020. How context affects lan- guage models' factual predictions. AKBC.
Yuqi Zhu. Xiaohan Wang. Jing Chen, Shuofei Qiao, Yixin Ou, Yunzhi Yao, Shumin Deng, Huajun Chen, and Ningyu Zhang. 2023. Llms for knowledge graph construction and reasoning: Recent capabilities and future opportunities. arXiv.
Fabio Petroni, Tim Rocktäschel, Sebastian Riedel, Patrick Lewis, Anton Bakhtin, Yuxiang Wu, and Alexander Miller. 2019. Language models knowl- edge bases? In EMNLP.

---

A Predictions
In the following we show some predictions from different experimental setups. Q marks the
A.1.2 GPT-4
query, P the prediction by GPT-3 and the ground truth object. Please note that prompt GPT-3, relation-specific examples are used at the beginning each query. The prompts w/ examples can be seen in Appendix B.
Q: who are the developers of The Incredible Hulk? Give me a list with no additional text.
P: Edge of Reality
A: Radical Entertainment, Edge of Reality, Amaze Entertainment, Pocket Studios, Universal Inter- active
A.1 Standard
This section shows predictions from our standard setup with the number of few-shot examples being
8. The prompts B.I.
A.1.1 GPT-3
Q:In which languages is Archi- tecture and secondary structure of entire HIV-1 RNA genome available? Give no additional text.
Q: As It Is in Heaven original language
P: Swedish
list with
A: Swedish
P: English
A: Im sorry, but couldn't find specifie fic information about the languages in which the mentioned scientific study or information is available.
Q: John Gruber employer
P: Daring Fireball
A:
The Pittsburg Times, Pitts- burgh Post, Pittsburgh Gazette, The Gazette Times, Pittsburgh Pi- rates
A.1.3 ChatGPT
Q: Who are the developers of The Incredible Hulk? Give me a list with no additional text.
Q: Hillary Canyon continent
P: Antarctica
A: North America
P: Edge of Reality
A: Edge of Reality, Amaze Enter- tainment, Beenox, Foundation 9 Entertainment, Griptonite Games, Fizz Factor, Nucore
Q: The Incredible Halk devel-
oper
P: Edge of Reality
A: Edge of Reality, Amaze Enter- tainment
Q:In which languages is In which languages is Architecture and secondary structure of an entire HIV-1 RNA genome available? Give me list with no additional text. available? Give me list with no additional text.
Q: Architecture and secondary structure of an entire HIV-1 RNA genome # language of work P: English
P: English
A: English
A: English

---

A: Kingdom of Italy, Italy
A.2 Textual context
Q: Luis Alfredo López # citizen- ship
In this section we show some predictions from our standard setup with textual contexts. The examples used for prompting in this setup can be seen in section B.2.
C: Congratulations to Luis "Al- fredo" Lopez, associate from Vistar Southern California, for being inducted into the 2020 IFDA Truck Driver Hall of Fame P: Mexico
Q: Neko-Nin Nin exllarr t developer Developer.
A: Honduras
C: Apr 21, 2017 Whirlpool : Publisher. Sekai
Project ; Released. Apr 21, 2017 : 0S: Windows 7 or above ; Pro- cessor: 1.2 GHz Pentium
A.3 Don't Know
In this section we present some predictions from our setup using Don't know prompting. The exam- ples used for prompting in this setup be seen in section B.3.
P: Whirlpool, Sekai Project
A: Whirlpool
Q: Navel*Plus developer
Q Visvanath Kar # native lan- guage
C: Position: Qlik Sense Devel- oper. Location: Phoenix, A2 SSIS and Datastage are a plus. Thanks & Regards,, Sravan Kumar Navel Technologies Inc.
P: Don's t. know
A: Odia
P: Navel Technologies Inc.
Q: Eikeviken # manufacturer
A: Navel
P: Don't know
A: Samsung Heavy Industries
Q: Andrei Krasilnikov native language
Q: Volkwin Marg # work location
C: Languages Russian. Native or bilingual proficiency En- glish. Full professional profi- ciency French. Limited working
P: Hamburg, Aachen
A Berlin, Brunswick, Hamburg, Aachen
proficiency . Spanish. Limited working
Q: Elvive # manufacturer
P: Spanish, French, English, Rus-
P: L L'Oréal
sian
A: L' L'Oréal
A: Russian
A.4 Out-of-KB facts
Q: Sergio Aquilante citizen- ship
In this section we present some predictions for out- of-KB facts evaluated in manual evaluation. For prompting we used the examples from our standard setup B.1.
C: View Ariana Morais' profile on LinkedIn, the world's largest professional community, Ariana has 1 job listed on their profile. See the complete profile on P: Brazil
Q: Lalith Perera # citizenship
P: Sri Lanka

---

Q: Chronicles of Mystery: The
Scorpio Ritual developer A: Games S.A.
Q: The Splatters developer
P: SpikySnail Games
Q: Call of Duty 3 # developer
A: Treyarch # Exakt Entertain-
ment
Q: Perú Negro location for-
Q: Samplitude # developer
mation
A: Bellevue Investments
P: Lima
Q: Dangun Feveron # developer
A: CAVE
Q: Sega Classics Arcade Collec-
Q: Albrecht Quadt languages spoken, written or signed in P: German
tion developer
A: Sega
Q: Python # developer
A: Python Software Foundation
Guido Rossum
Q: Wreckless: The Yakuza Mis-
Q: Allen Coral Atlas # developer
sions # developer P: Bunkasha Games
A: Vulcan Inc. University
Queensland Carnegie Institution for Science
Q: Avatar: The Last Airbender
Into the Inferno developer
B Prompts
A: Nickelodeon
B.1 Standard
nativeLanguage
employedBy
Q: Bill Byrge # native language
A: English
Q: Silvestre Paredes employer
Q : Augustin Michel # native lan-
A: Universidad Politécnica de
guage
Cartagena
A: French
Q: Masashi Kamogawa employer
Q: : Ingrian Finns # native lan-
A: Waseda University
guage
Q: Ana Rosa Rama Ballesteros employer
A: Finnish & Russian
Q: Nanu Ram Kumawat # native lan-
A: University of Jaén # Univer-
guage
sity of Jaén
A: Rajasthani & Hindi
Q: Sara Akbar employer
Q: Jumber Dochviri # native lan-
A: Kuwait 0il Company
guage
Q: Pius
employer
A: Georgian
A: University of Bologna # Uni-
Q: Rosa Estarås #native language
versity of Pavia
A: Spanish # Catalan
Q: Masao Kotani employer
Q: Vladimir Fotievich Kozlov #
A: University of Tokyo # Osaka
native language
University
A: Russian
Q: Bernadeta Patro Golab em-
Q: Vladimir Nemkin # native lan-
ployer
guage
A: Medical University of Warsaw
A: Russian Ukrainian
Q: Andy Hertzfeld employer
A: Google
B.2 Textual context
developedBy
inContinent

---

Reventador continent C: Daily explosions, ash plumes, lava flows, and incandescent block avalanches during February- July 2022. Volcán gl Reventador is located 100 km E of the main... Reventador is an active strato- volcano which lies in the eastern Andes of Ecuador. It lies in remote area of the national park of the same name, which is... A: South America t Americas
Q:
nent C: The St. Lawrence River is large river in the middle lat- itudes of North America. Its headwaters begin flowing from Lake Ontario in roughly north- easterly... St. Lawrence River, hydrographic system of east- central North America. the outflow Lake Ontario and leads into the Atlantic Ocean in the...
starts
Q: Fatimid Caliphate continent C: The Fatimid Caliphate was an Ismaili Shi'a caliphate extant from the tenth to the twelfth Fatimid Caliphate is located in Continental Asia. Encompassing the vast Sahara in North Africa, alongside the Levant in the Mid- dle East, the entirety of the Caliphate consists of Arid biome, save for Sicily and...
A: North America
Q: Cerro El Charabón # continent
C: 65, Estancia El Charabón. 49.
66, Área costero-marina Cerro Verde Islas de la Coronilla- Årea General. 48. 67, Area protegida Laguna de Castillos Tramo... Casa del Sol Boutique Hotel. A cozy stay awaits you in Machu Picchu. San Pedro de Atacama
Altiplánico Welcome to E1 Charabon. El Charabon.
A: Asia # Africa # Europe
Q: Cerro Tenán continent C: See photos, floor plans and more details about 262 S Paseo Cerro in Green Valley, Arizona. Visit Rent. now for rental rates and other information about this... 200 N Continental Blvd NEC Market St & Via Cerro Investment Services Landlord Representation Tenant Represen- tation Industrial: Warehouse
A: Americas
Q: Hinterer Seekopf # continent
C: Following the breakup of Pangea during the Mesozoic era, the continents of ... of the best day hikes in Kalka- 1pen National Park is the Ho- her Nock Seekopf. Dec 5, 2016 ... Hinterer Steinbach. In- haltsverzeichnis aufklappen Inhaltsserrzeichnis einklappen Charakteristik. Hinweise; Sub-
8...
A: Americas
jektive Bewertung...
Q: Mississippi River continent The Missis- sippi River is the second longest river in North America, flowing 2, 350 miles from its source at Lake Itasca through the center of the...
A: Europe
C: Feb 10, 2022
Q: Šembera # continent
C: Rephrasing Heidegger: A Con- panion to Heidegger's Being and Time (Sembera, ... Being and Time (Suny Series in Contemporary Continental Philosophy). Feb 26, 2016 ... Coming from Uganda, UNV PO Flavia Sembera was famil- iar with diversity.
or about one-eighth of the entire continent. The Mississippi River lies entirely within the United States. Rising in Lake Itasca in Minnesota, it flows...
shared across the continent while expe- riencing Zambia's beautiful...
A: Americas # North America
A: Europe
Q: St. Lawrence River conti-

---

work location
11, 2015 Ole Olsen Bak eller Ormhaug som 1803 kjøpte girden ØvreViken 461, fra 1819 var Ole Olsen Teslien. someide Nedre n 8 1i,...
Q: Karel Lodewijk Sohl work location
Julius Sand.
C: Karel Lodewijk SOHL geboren op
18 februari 1895 te Maastricht.
A: Oslo
Hij huwde Johanna Catharina Maria VAN BINSBERGEN 20 november
Q: John Sewell # work location C: John W Sewell was born in
1924 te Roermond. Karel LANOO.
1867 in Elbert County, Georgia, and moved with his parents Florida when was 19 years old. Sewell, working for Henry Fla- gler,
Karen ANDERSON. Karina MROB.
Karine LALIEUX ... Lena SOHL. Leonidas MAKRIS. Leopold SPECHT Lodewijk ASSCHER. Lora
Implementation Consulting Learn more about John Sewell, CMRP's work experi- ence, education, connections & more by visiting their profile LinkedIn.
LYUBENOVA. Loren LANDAU
A: Maastricht # Paris # Maas- tricht
Q: H.G. Broekhuisen work location
C: Nov 28, 2018 ... Fourth, we are grateful for the work of the e-Vita platform helpdesks: Mireille
A: London London
Q: Alan R. Battersby # work loca-
tion
Van Spall HG, Rah-
C: Sir Alan Rushton Battersby FRS
man T, Mytton o, Ramasundara- hettige C,... Apr 30, 2013
(4 March 1925 - 10 February 2018)
was an English organic chemist best known for his work to de- fine the chemical intermediates... Dec 9, 2018 R. Hattersby, University of Cam- bridge Cambridge, United Kingdom. "for their fundamental contri- butions to the elucidation of the...
Saskia F van Vugt, gen- eral practitioner 1,; Berna L Broekhuizen,
In Flanders (Belgium) this work was supported by the Research...
Battersby. Alan
A: Makassar
Q: Hermann Kretzschmer work lo- cation
C: The author died in 1890, this work is in the public do- main in its country of origin and other countries and areas where the copyright term is the s... Stay up to date with Hermann Kretzschmer (German, 1811 1890) Discover works for sale, auction results, market data, news and exhibitions Mu- tualArt.
A: Cambridge
Q: A. Kate Miller # work loca- tion
C: Title/Position. Advocacy Di- rector. Department. Advocacy Department. Pronouns. she Kate works in a number of issue areas and is always seeking com- mon ground...
Miller's work experience, education, connec- tions more by visiting their profile on LinkedIn.
A: Berlin # Düsseldorf
Work
Q: Ole Olsen Teslien work loca- tion
location: Chicago, Illinoi, United States. Work... A: Indianapolis
C: og Bjom Pedersen Teslien 1831
- kjøpesun 3500 spd. Fra 1845 var Bjom Ptdersen alene eier. Efter Cle Jensen overtok sønnen Anders Olsen g. i 1850. Jul
Q: Andrzej Grzesik # work loca-
tion
C: View Andrzej Grzesik's pro-
file LinkedIn, the world's

---

largest professional community. Andrzej has 15 jobs listed on their profile. Nov 14, 2016 Congratulations to the newest Java Champion Andrzej Grzesik! in Poland (sfi.org.pl) and in his work as Sun Campus Am- bassador.
A: English
Q: Ingeborg Heintze # Languages spoken, written or signed
A: Don't know
Q: Alicia Coduras Martinez # lan-
guages spoken, written or signed
A: Catalan # Spanish
Q: Adam Budak # languages spoken,
A: Warsaw
written signed
A: Don't know
B.3 Don't Know
producedBy
Q:
Strati manufacturer
A: Local Motors # Oak Ridge Na-
tional Laboratory
Q: Philips VG-8235 manufac-
turer
A: Don't know ss America (1939)
# manufacturer
A: Newport News Shipbuilding
Q: HMCS Brandon manufacturer
A: Don't t know
Q: Cluster Platform 3000 SL1602,
Xeon L55xx 2.26 GHz, GigE & manu facturer
A: Hewlett Packard Enterprise
Q: FOWER CHALLENGE manufac-
turer
A: Don't know
Q: Suzuki Katana manufacturer
A: Suzuki
Q: German submarine U-1223 man
ufacturer
A: DDnn' t know
spokenLanguage
Q: Allen G. Thurman languages spoken, written or signed A: English
Q: Rifat Hairy languages spo-
ken, written or signed
A: Don't know
Q: Izabela Filipiak languages spoken, written or signed
A: American English # Polish
Q: Vasudev Gopal Paranipe lan-
guages spoken, written or signed A: Don't ' know
Q: Jonathan M. Katz languages
spoken, written or signed


Reviews:
[{"review_id": "rRjeODkMqd", "reviewer": "Reviewer_QRC9", "summary": "", "questions": "Question A: It is not clear to me how someone would use GPT-3 to get high precision outcome. Is it the case that we can prompt GPT-3 and based on a predefined threshold to filter out predictions so that we may get roughly high precision results as mentioned in this paper? If so, make it clear in the paper. \n\nQuestion B: Then, do we have to define a threshold for each and every relation in the KG?\n", "limitations": "", "rating": 4, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your thoughtful comments.\n\nReply to questions:\n\nA. If the goal solely is to achieve high precision, we would sort GPT-3 outputs by confidence scores with cut-off at some threshold. However, there is no easy way to aim for maximal recall at sufficiently high precision. The point of our paper is that this realistic goal for KB completion poses a major challenge.\n\nB. Yes, such thresholds should be specifically chosen for each relation. Using withheld ground-truth data, we can automatically compute these per-relation thresholds such that the model yields a desired target precision (presumably at limited recall, though).", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "•\tPaper is easy to understand and provides a timely analysis of GPT-3 capabilities on KG completion task (how useful it would be)\n\n•\tFindings presented in the paper may be of interest to a broader community.", "reasons_to_reject": "•\tIt is not clear how someone would use the insights presented in the paper to get high precision results (also see questions below). ", "questions_to_authors": "Question A: It is not clear to me how someone would use GPT-3 to get high precision outcome. Is it the case that we can prompt GPT-3 and based on a predefined threshold to filter out predictions so that we may get roughly high precision results as mentioned in this paper? If so, make it clear in the paper. \n\nQuestion B: Then, do we have to define a threshold for each and every relation in the KG?\n", "paper_topic_and_main_contributions": "The paper presents an evaluation of GPT-3 on the potential of its use for KB completion. They used a recent KB completion benchmark dataset to prompt GPT-3 to get the relations generated. The work is more of an evaluation setup to see how good GPT-3 can be for the KB completion task. I think this analysis may provide good insights to works around KGs and LLMs, especially LLMs’ capabilities on knowledge related tasks. ", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "0Fq92wbKDm", "reviewer": "Reviewer_5ndf", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 5, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your thoughtful comments.\n\nReply to questions:\n\n1. We performed additional experiments using i) the smaller GPT-3 text-curie-001 with 6B parameters, and ii) GPT-4. As expected, recall@precision90 drops drastically for the smaller GPT-3 model (e.g., from 0.76 to 0.39 for writtenIn, from 0.39 to 0.16 for ownedBy etc.). For GPT-4, the proper evaluation is difficult as its API does not yield confidence scores. Instead, we simply evaluate the overall precision per relation. Some relations are handled very well (e.g., 0.85 for writtenIn, 0.74 for ownedBy), but others still show poor results (e.g., 0.31 for foundedIn, 0.29 for playsInstrument), and the overall recall varies between 30 and 80 percent. The table below shows full results. If accepted, we will include these latest results in the paper.\n2. The performance does depend on the size and quality of the pre-training corpus. As for the influence of size, there is also prior/parallel work; see, for example: \nN. Kandpal et al.:  Large language models struggle to learn long-tail knowledge. ICML, July 2023.\nEvaluating the influence of the pre-training content quality poses major difficulties, though, as the most interesting GPT models do not convey specifics of their pre-training data. We plan to pursue studies with open-data models such as BLOOM. This is future work.\n3. Reproducibility: \nWe use a random split of the publicly available dataset WD-Known [1]. We will publish full details as well as Mturk-based human judgements upon acceptance. For reviewer access, we have created a preliminary, anonymous repository at https://drive.google.com/drive/folders/1LNv6k7-TzoqzZfgsHxoqbC6m-jV75USW?usp=sharing\n\n    [1] Veseli et al. : Evaluating Language Models for Knowledge Base Completion. European Semantic Web Conference (ESWC) 2023, data repository: https://github.com/bveseli/LMsForKBC     \n\n| Relation        | GPT-3 davinci $R@P95$ | GPT-3 davinci $R@P90$ | GPT-3 curie $R@P95$ | GPT-3 curie $R@P90$ | BERT$_{Large}$ $R@P95$ | BERT$_{Large}$ $R@P90$ | GPT-4 $Precision$ | GPT-4 $Recall$ | GPT-3 davinci $Precision$ | GPT-3 davinci $Recall$ |\n|-----------------|---------------------------------|---------------------------------|---------------------------------|---------------------------------|-------------------------|-------------------------|------------|------------|------------------------------|------------------------------|\n| writtenIn       | 0.69                            | 0.76                            | 0.2                             | 0.39                            | 0                       | 0                       | 0.85       | 0.81       | 0.91                         | 0.78                         |\n| ownedBy         | 0.37                            | 0.39                            | 0.11                            | 0.16                            | 0                       | 0                       | 0.74       | 0.54       | 0.6                          | 0.44                         |\n| nativeLanguage  | 0.22                            | 0.7                             | 0.11                            | 0.6                             | 0.43                    | 0.58                    | 0.87       | 0.87       | 0.69                         | 0.86                         |\n| LanguageOfFilm  | 0.21                            | 0.33                            | 0                               | 0.01                            | 0                       | 0.01                    | 0.78       | 0.63       | 0.58                         | 0.52                         |\n| hasCapital      | 0.19                            | 0.31                            | 0                               | 0                               | 0.02                    | 0.02                    | 0.82       | 0.52       | 0.77                         | 0.44                         |\n| officialLanguage | 0.09                            | 0.24                            | 0                               | 0                               | 0.03                    | 0.25                    | 0.69       | 0.68       | 0.67                         | 0.64                         |\n| foundedIn       | 0.06                            | 0.08                            | 0                               | 0                               | 0                       | 0                       | 0.31       | 0.6        | 0.4                          | 0.38                         |\n| playsInstrument | 0.02                            | 0.02                            | 0                               | 0                               | 0                       | 0                       | 0.29       | 0.41       | 0.16                         | 0.18                         |\n| partOf          | 0.01                            | 0.01                            | 0                               | 0                               | 0                       | 0                       | 0.3        | 0.58       | 0.17                         | 0.1                          |\n| citizenOf       | 0.01                            | 0.24                            | 0                               | 0                               | 0.02                    | 0.03                    | 0.75       | 0.65       | 0.67                         | 0.6                          |\n| spokenLanguage  | 0.01                            | 0.47                            | 0.02                            | 0.13                            | 0                       | 0.24                    | 0.57       | 0.73       | 0.54                         | 0.76                         |\n| playerPosition  | 0.01                            | 0.02                            | 0                               | 0                               | 0                       | 0                       | 0.45       | 0.45       | 0.18                         | 0.24                         |\n| inContinent     | 0.01                            | 0.01                            | 0                               | 0                               | 0                       | 0                       | 0.8        | 0.72       | 0.61                         | 0.6                          |\n| namedAfter      | 0.01                            | 0.07                            | 0.01                            | 0.01                            | 0                       | 0                       | 0.68       | 0.67       | 0.53                         | 0.44                         |\n| hostCountry     | 0.01                            | 0.34                            | 0.02                            | 0.02                            | 0                       | 0                       | 0.83       | 0.54       | 0.75                         | 0.48                         |\n| musicLabel      | 0.01                            | 0.02                            | 0                               | 0                               | 0                       | 0                       | 0.65       | 0.68       | 0.16                         | 0.18                         |\n| hasReligion     | 0                               | 0.02                            | 0                               | 0                               | 0.03                    | 0.07                    | 0.54       | 0.82       | 0.47                         | 0.38                         |\n| developedBy     | 0                               | 0.11                            | 0                               | 0                               | 0.04                    | 0.04                    | 0.51       | 0.83       | 0.45                         | 0.5                          |\n| countryOfJurisdiction | 0                        | 0.08                            | 0.05                            | 0.08                            | 0.02                    | 0.02                    | 0.37       | 0.23       | 0.52                         | 0.22                         |\n| subclassOf      | 0                               | 0.37                            | 0                               | 0                               | 0                       | 0                       | 0.16       | 0.31       | 0.73                         | 0.85                         |\n| diplomaticRelation | 0                             | 0.01                            | 0                               | 0                               | 0                       | 0                       | 0.41       | 0.7        | 0.7                          | 0.17                         |\n| CountryOfOrigin | 0                               | 0.01                            | 0                               | 0                               | 0.01                    | 0.01                    | 0.66       | 0.38       | 0.48                         | 0.31                         |\n\n**Table**: In this table, GPT-3 curie refers to GPT-3 text-curie-001, and GPT-3 davinci refers to GPT-3 text-davinci-003.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The paper is well written and easy to understand. The authors clearly justify their study based on previous studies that do not pay attention to the long-tail when extending KBs.\n\nThe authors also provides a study per relation and using a different prompting.", "reasons_to_reject": "I only miss experiments with another LLMs more similar to GPT-3 than Bert-Large (the other model used in this paper). For example, the authors could test an smaller  GPT-3 and study results.  Besides, I guess that the results depends not only on the size of the model, but also on the pre-training documents. I think the authors should study this.", "questions_to_authors": "", "paper_topic_and_main_contributions": "This is a study about the use of GPT-3 for KB completition. The novelty of this study is the fact of paying attention to non-popular statements, what allows to extend KBs in contrast to previous studies. \n\nThe main contributions are:\n- a study about KB completition on new facts instead of facts already in the KB. This is important to really extend KBs.\n- evaluate recall at different precision levels\n\nThe paper is well written and easy to understand. I only miss experiments with another LLMs more similar to GPT-3 than Bert-Large (the other model used in this paper). For example, the authors could test an smaller  GPT-3 and study results.  Besides, I guess that the results depends not only on the size of the model, but also on the pre-training documents. I think the authors should study this.", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Typos_Grammar_Style_and_Presentation_Improvements": "There is a missing reference in line 138"}, "extra_text": {}}, {"review_id": "8yD3tyRmjM", "reviewer": "Reviewer_Fktf", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your thoughtful comments. If the paper gets accepted, we would use the additional 5th page to expand on the methodological choices and result discussion.\n\nClarification of experimental setup and reproducibility: \n\nWe use a random split of the publicly available dataset WD-Known [1]. We will publish full details as well as Mturk-based human judgements upon acceptance. For reviewer access, we have created a preliminary, anonymous repository at https://drive.google.com/drive/folders/1LNv6k7-TzoqzZfgsHxoqbC6m-jV75USW?usp=sharing\n\n[1] Veseli et al. : Evaluating Language Models for Knowledge Base Completion. European Semantic Web Conference (ESWC) 2023, data repository: https://github.com/bveseli/LMsForKBC", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The article deals with a topic of certain and current interest on how to use large language models for the construction and expansion of graphs.", "reasons_to_reject": "I do not find the paper informative enough. For instance, some methodological choices are not well-motivated or explained (apart from the reference) and some results need further testing or explanation (e.g., the lowest scores reached for some of the relations).", "questions_to_authors": "", "paper_topic_and_main_contributions": "This paper presents an experiment for knowledge base completion by means of GPT-3. \nThe proposed experiment focuses on the prediction of 41 relations from Petroni et al. (2019). Different promptings have been tested, including a few-shot one to produce a ‘don’t know’ answer. The evaluation has been performed automatically and manually and shows some interesting results on the topic.", "excitement": 3, "reproducibility": 2, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {"Typos_Grammar_Style_and_Presentation_Improvements": "Line 054, please use \\textcite \nLine 130, please use \\textcite\nLine 138, please, check the missing reference\nLine 141, The → the\nLine 199, please use \\textcite \nLine 206, please use \\textcite \nCheck the reference format"}}]

Author rebuttal:

Meta-review: The paper investigated the potential of using LLMs (GPT-3) for KB completion. The paper is easy to understand and provides a timely analysis of LLMs' capabilities on KG completion tasks. Findings presented in the paper may be of interest to a broader community. The authors clearly justify their study based on previous studies that do not pay attention to the long tail when extending KBs. The authors also provide a study per relation and use a different prompting. Overall, it is a solid short paper presenting interesting findings.
Decision: Accept-Findings