SlowGuess/NAACL_2021_OCR huggingface dataset:
Kurt Shuster, Da Ju, Stephen Roller, Emily Dinan, Y-Lan Boureau, and Jason Weston. 2019. The dialogue dodecathlon: Open-domain knowledge and image grounded conversational agents. arXiv preprint arXiv:1911.03768.
Mihai Surdeanu. 2013. Overview of the tac2013 knowledge base population evaluation: English slot filling and temporal slot filling. In TAC.
Mihai Surdeanu and Heng Ji. 2014. Overview of the english slot filling track at the tac2014 knowledge base population evaluation. In Proc. Text Analysis Conference (TAC2014).
Alon Talmor and Jonathan Berant. 2018. The web as a knowledge-base for answering complex questions. Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers), pages 641-651.
James Thorne and Andreas Vlachos. 2020. Avoiding catastrophic forgetting in mitigating model biases in sentence-pair classification with elastic weight consolidation. arXiv preprint arXiv:2004.14366.
James Thorne, Andreas Vlachos, Christos Christodoulopoulos, and Arpit Mittal. 2018a. FEVER: a large-scale dataset for fact extraction and verification. In NAACL-HLT.
James Thorne, Andreas Vlachos, Oana Cocarascu, Christos Christodoulopoulos, and Arpit Mittal. 2018b. The fact extraction and verification (fever) shared task. EMNLP.
James Thorne, Andreas Vlachos, Oana Cocarascu, Christos Christodoulopoulos, and Arpit Mittal. 2019. The fever2.0 shared task. In Proceedings of the Second Workshop on Fact Extraction and VERIFICATION (FEVER), pages 1-6.
Alex Wang, Yada Pruksachatkun, Nikita Nangia, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel Bowman. 2019. Superglue: A stickier benchmark for general-purpose language understanding systems. In Advances in Neural Information Processing Systems, pages 3261-3275.
Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R Bowman. 2018. Glue: A multi-task benchmark and analysis platform for natural language understanding. arXiv preprint arXiv:1804.07461.
Thomas Wolf, L Debut, V Sanh, J Chaumont, C Delangue, A Moi, P Cistac, T Rault, R Louf, M Funtowicz, et al. 2019. Huggingface's transformers: State-of-the-art natural language processing. ArXiv, abs/1910.03771.
Tomer Wolfson, Mor Geva, Ankit Gupta, Matt Gardner, Yoav Goldberg, Daniel Deutch, and Jonathan Berant. 2020. Break it down: A question understanding benchmark. Transactions of the Association for Computational Linguistics, 8:183-198.
Ledell Wu, Fabio Petroni, Martin Josifoski, Sebastian Riedel, and Luke Zettlemoyer. 2019. Zero-shot entity linking with dense entity retrieval. arXiv preprint arXiv:1911.03814.
Wenhan Xiong, Jingfei Du, William Yang Wang, and Veselin Stoyanov. 2019. Pretrained encyclopedia: Weakly supervised knowledge-pretrained language model. arXiv preprint arXiv:1912.09637.
Deshraj Yadav, Rishabh Jain, Harsh Agrawal, Prithvijit Chattopadhyay, Taranjeet Singh, Akash Jain, Shiv Baran Singh, Stefan Lee, and Dhruv Batra 2019. Evalai: Towards better evaluation systems for ai agents. arXiv preprint arXiv:1902.03570.
Zhilin Yang, Peng Qi, Saizheng Zhang, Yoshua Bengio, William Cohen, Ruslan Salakhutdinov, and Christopher D. Manning. 2018. HotpotQA: A dataset for diverse, explainable multi-hop question answering Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 2369-2380.
Wikipedia Representation We represent the KILT knowledge source as a collection of JSON records, one per Wikipedia page. Each record is assigned: (i) a unique Wikipedia id; (ii) a unique Wikipedia title; (iii) a text field containing a list of strings, one for each paragraph, bulleted list item, and section header (for which we preserve the hierarchical structure); (iv) a list of anchors elements, one for each hyperlink in the original page text, with span reference in the text field and page linked; (v) a list of categories; (vi) a url redirecting to the original html for the page, with timestamp of the last page revision before the considered snapshot.
Datasets Mapping Details In  $FEVER$ , often multiple pieces of knowledge must be combined to produce an output. For example,  $30\%$  of claims have more than one equally-valid provenance and  $16\%$  require the combination of multiple evidence spans. The second iteration (FEVER2.0, Thorne et al., 2019) introduces a collection of adversarial instances. For KILT, we merge the two versions of  $FEVER$  into a single resource and consider only supported refuted claims. We exclude all claims classified as not having enough information since these instances have no evidence to assess the claim and cannot be mapped to the KILT knowledge source. Therefore we cannot assess whether such label is still appropriated given our snapshot. Moreover, we design KILT as an in-KB resource where each instance can be answered and corroborated by information in the knowledge source.
In the Zero Shot RE dataset a set crowd-sourced template questions are defined for each relation — for example, What is Albert Einstein's alma mater?. Each datapoint reports a Wikipedia sentence expressing the fact that we take as provenance. Some examples in the dataset are negative, obtained by matching a valid question and a random sentence, that likely does not contain the answer. To consider an open-domain version of this dataset and align the input/output with the KILT interface we reformatted this dataset, as follows: (i) exclude negative pairs - since we consider the whole knowledge source (as opposite to a single sentence) as text all questions can be answered; (ii) group template questions by the subject-relation pair, and create a single datapoint for each (input as above); (iii) randomly split the set of relations, in line with the original dataset, into three disjoint sets train (with
84 relations), dev (12 relations) and test (24 relations)—systems are tested on relations never seen during training; (iv) use the subject entity as the query against Wikipedia titles for the first step of the mapping strategy, and (v) include all template questions in a meta field.
For  $T$ -REx, We filter out facts with more than 20 provenances, relations with less than 1000 facts, and merge all the facts for the same subject-relation pair (i.e., for 1-N and M-N relations there could be multiple valid answers), resulting in 113 relations and 2.3M facts. We include object aliases as equally valid answers and report in a meta field subject aliases as well as all surface mentions for the subject, relation and object. We randomly select 5k facts for both dev and test set.
To define an open-version of the Natural Questions dataset we follow Lee et al. (2019) and (1) keep only questions with short answers and (2) discard all answers with more than five tokens.
To find answers in TriviaQA, the original work used distant supervision: (1) find Wikipedia entities in the question with the TAGME entity linked (Ferragina and Scaiella, 2011); (2) search for the answer (and all Wikipedia aliases) in the corresponding page; (3) if the answer is found, add the page in the evidence documents. Therefore, the documents are not guaranteed to contain evidence for the question-answer pair (but the authors estimate that they do  $79.7\%$  of the time).
In ELI5 Evidence documents are automatically gathered, and we focus on the case where evidence documents are extracted from Wikipedia. However, as the original work first collected question-answer pairs from the subreddit Explain Like I'm Five, the documents are not guaranteed to contain evidence.
For Wizard of Wikipedia we discard cases where the dataset does not contain provenance. Moreover, we consider a full open-domain setting where no topic is provided for the conversation and the model must search over all of Wikipedia for knowledge at each dialogue turn (rather than the provided knowledge candidates for each turn in the original dataset). We use the unseen split for dev and test.
Performance Impact Of The Mapping Strategy We want to assess if the performance we obtain after mapping each dataset to a unified Wikipedia snapshot are in line with what reported in previous work. Thorne and Vlachos (2020) report a 2-way accuracy of 79.09 for the FEVER dev set when considering purely claims in input to a RoBERTa
Figure 2: Number of pages vs number of dataset with knowledge in a page. 1,642,311 pages contains knowledge needed for KILT ( $\sim$ 28% of the knowledge source).
based classifier (Liu et al., 2019). Our dev set includes also the adversarial examples of FEVER 2.0, nevertheless the performance of BART are in line (80.67 dev, 78.93 test). Karpukhin et al. (2020) report 41.5 for EM on the open domain version of the NQ dev set<sup>18</sup>. With our setting, DPR achieves an on-par performance on the dev set, with a 42.58 EM (50.43 F1-score). Results on our brand new NQ test set are 3/4 points lower for EM and F1-score than dev results. We don't evaluate multi-hop specific baselines on KILT but the current best F1-score for HotpotQA is 75.43 according to the official leadearboard<sup>19</sup>, that is quite far from what achieved by our general solutions. BLINK results are in line with what reported in the GitHub repository<sup>20</sup> for all three entity linking datasets. The Tranformer MemNet of Dinan et al. (2019) achieves a F1-score of 14.3 on the original version of the WW dataset while 11.5 in our setting, probably because in KILT we consider an harder open-domain setting.
Retrieval Baselines The ability to retrieve relevant documents from Wikipedia given an input is an important aspect we assess in KILT. A system should select only the relevant knowledge needed for the task, without redundant or excess information. A way to surface such knowledge is using a dedicated retrieval system. We consider three off-the-shelf retrievers and investigate drastically different retrieval paradigms: (i) Tfidf with the DrQA Document Retriever (Chen et al., 2017)—traditional page-level sparse vector space retrieval model; (ii) DPR (Karpukhin et al., 2020)—a modern passage-level retrieval solution using dense rep
resentations; (iii) A combination of BLINK (Wu et al., 2019) and flair (Akbik et al., 2019)—retrieval solution that ranks pages according to entities in the input.
The DrQA Document Retriever combines bigram hashing and TF-IDF matching to return relevant Wikipedia pages given an input. DPR splits each Wikipedia page into disjoint 100-word passages $^{21}$  and encodes passages and inputs with a BERT-based bi-encoder to perform dense Maximum Inner Product Search. The BLINK entity linking system uses a BERT-based bi-encoder to encode each Wikipedia page as well as each input, where a single entity mention is tagged. Final results are refined with a BERT-based cross-encoder. To use BLINK for retrieval, we look for entity mentions in each input with flair, then use BLINK to return a ranked list of Wikipedia pages for each entity mention. When multiple entities are identified in the input, we merge results and sort by score. The input string might not contain tags. For all systems, we use the index created on the KILT knowledge source.
We also experiment with multi-tasking, by jointly training a single DPR model on all KILT training data. We use uniform sampling to balance the datasets. In particular, the Multi-task variant of DPR is a single dense passage retriever, trained jointly on the union of TQA, NQ, HoPo, FEV, zsRE, AY2, T-REx and WoW. In order to avoid large datasets, such as T-REx, from having an oversize effect, we resample all datasets uniformly, such that every training epoch contains 150k samples from each task. Batches are formed from a single dataset at a time, iterating through the various datasets in a round-robin fashion.
Task-specific Baselines Approaches to the KILT Benchmark should be able to generalize to many different tasks, as developing model architectures that can represent knowledge generally is a valuable direction. However, several tasks may benefit from dedicated architectures designed for them.
For fact checking, we consider NSMN (Nie et al., 2019), the highest scoring system from the FEVER shared task (Thorne et al., 2018b). We use the public model[22] pre-trained on FEVER, and consider not enough information predictions as false.
2122,220,793 passages in the KILT knowledge source. Following Karpukhin et al. (2020) we don't consider Wikipedia bulleted lists in the text.
22available at https://github.com/easonnie/combine-FEVER-NSMN
18Reported as test results in (Karpukhin et al., 2020)
19https://hotpotqa.github.io
20 https://github.com/facebookresearch/ BLINK
Moreover, we develop a fact checking baseline that combines a BERT-base classifier with passages returned from DPR where the claim and retrieved passage are input. The classifier is trained to label the claim-passage pair as supported or refuted with an additional neutral class for negative-sampled unrelated passages. Unrelated passages are sampled from two sources: (1) DPR-retrieved passages from pages that are not in the list of pages in the instance's provenance and (2) passages sampled uniformly at random from pages in the instance's provenance. At inference, we classify the first sentence of the Wikipedia pages retrieved by the top-100 DPR passages against the claim. Using pages labelled as supported or refuted, we label the claim through majority voting. For claim provenance, we re-rank passages by probability according to this label.
For Open Domain QA and Slot Filling, we use DPR combined with the pre-trained BERT-based extractive reading comprehension model of Karpukhin et al. (2020). We use the model pretrained on TriviaQA for HotpotQA and the model pre-trained on Natural Questions for Zero Shot RE. We reduce the slot filling problem to question answering, by using the specified template questions. We consider a single random template question per subject-relation during inference.
For Entity Linking, we consider BLINK.
For Dialogue, we consider the Generative Transformer MemNet (Dinan et al., 2019) that encodes the dialogue history and knowledge to generate the next utterance. We use the pre-trained version available in ParlAI (Miller et al., 2017). Finally, to test the performance of combining BART and DPR on FEVER, we develop a classifier that uses these—full description in the appendix.
General Baselines A main motivation of the KILT Benchmark is to enable a unified approach towards a wide range of knowledge-intensive tasks. We analyze existing general architectures that can be used as a baseline for multiple tasks in KILT.
Large pre-trained sequence-to-sequence models such as BART (Lewis et al., 2019) and T5 (Raffel et al., 2019a) implicitly store a surprising amount of knowledge in their parameters (Petroni et al., 2019). We treat all KILT tasks as generative, relying on the knowledge accumulated by the model while pre-training, with no retrieval (similarly to Roberts et al. (2020)). We finetune pre-trained variants on all KILT tasks, using fairseq (Ott et al., 2019)
for BART and Huggingface's Transformer (Wolf et al., 2019) for T5.
A natural way to boost performance is to incorporate an explicit knowledge mechanism. For our BART+DPR baseline, we follow Petroni et al. (2020) to retrieve and comprehend the top-3 passages from DPR for each input sample and use context-enhanced training data to fine-tune a BART model. We use the DPR rank when reporting provenance for all except entity linking tasks. For entity linking, we report the Wikipedia id of the page whose title exactly matches the predicted string.
Recently, state-of-the-art results on a wide range of NLP tasks have been achieved by combining a trainable retrieval step with language modeling or generation (Guu et al., 2020; Lewis et al., 2020a). We experiment with fine-tuning RAG (Lewis et al., 2020b) on KILT tasks, establishing a strong baseline on all of them. RAG combines a DPR retriever with a BART generator, however, unlike in the case of our previous baseline, RAG back-propagates to the retriever's input encoder, learning to adapt the input embedding to retrieve more relevant results. At every generation step we retrieve top-5 passages and use them as provenance.
Dataset Label	Multi-hop	Average Provenance Size (APS)	Average Provenance Number (APN)	Average Provenance Pages (APP)	Average Answers Number (AAN)	Train Size	Dev Size	Test Size
FEV	x	1.12	1.35	1.13	1	104,966	10,444	10,100
AY2	1	1	1	1	18,395	4,784	4,463
WnWi	1	1	1	1	-	3,396	3,376
WnCw	1	1	1	1	-	5,599	5,543
T-REx	1	1.68	1.26	5.29	2,284,168	5,000	5,000
zsRE	1	1	1	1	147,909	3,724	4,966
NQ	1	3.22	1.57	2.08	87,372	2,837	1,444
HoPo	x	2.4	1	2	1	88,869	5,600	5,569
TQA	1	3.39	1.68	28.67	61,844	5,359	6,586
ELI5	1	1.21	1.18	4.69	272,634	1,507	600
WoW	1	1	1	1	94,577	3,058	2,944
Total	3,160,734	51,464	50,736
Table 6: Datasets statistics. APS refers to the average number of textual spans in each provenance set—for most of the datasets a single span is sufficient to provide enough evidence while FEV and HoPo might require more (hence they require multi-hop reasoning). APN indicates the average number of equally valid provenance sets for each instance while APP the average number of Wikipedia pages overall in the provenance (note that multiple spans might refer to the same Wikipedia page). Finally AAN reports the average number of equally valid gold answers per instance. We additionally report the size of the train, dev and test split for each dataset.
Figure 3: KILT datasets' interface. Each dataset is represented as a JSON Line file. The Figure shows the pseudo-JSON structure for each record in the files.
model	R-Precision	Recall@5	Accuracy	KILT-AC
test
BART	0.0	0.0	78.93	0.0
T5	0.0	0.0	76.3	0.0
NSMN	49.24	70.16	66.1	41.88
BART + DPR	55.33	74.29	86.74	47.68
RAG	61.94	75.55	86.31	53.45
BERT + DPR	72.93	73.52	69.68	58.58
dev
BART	0.0	0.0	80.67	0.0
BART + DPR	55.46	73.84	88.11	48.25
RAG	63.5	76.1	87.7	55.47
Table 7: FEVER
model	R-Precision	Recall@5	Accuracy	KILT-AC
test
RAG	72.62	72.62	72.62	72.62
T5	74.05	74.05	74.05	74.05
BART + DPR	75.49	75.49	75.49	75.49
BART	77.55	77.55	77.55	77.55
BLINK	81.54	94.73	81.54	81.54
dev
RAG	77.4	77.47	77.4	77.4
T5	81.84	81.84	81.84	81.84
BART	86.62	86.62	86.62	86.62
Table 8: AIDA CoNLL-YAGO
model	R-Precision	Recall@5	Accuracy	KILT-AC
test
BART + DPR	45.2	45.2	45.2	45.2
BART	45.91	45.91	45.91	45.91
T5	47.13	47.13	47.13	47.13
RAG	48.07	48.07	48.07	48.07
BLINK	80.24	91.47	80.24	80.24
dev
BART + DPR	44.96	44.96	44.96	44.96
T5	47.35	47.35	47.35	47.35
BART	47.91	47.91	47.91	47.91
RAG	49.0	49.0	49.0	49.0
Table 9: WNED-WIKI
model	R-Precision	Recall@5	Accuracy	KILT-AC
test
BART + DPR	46.87	46.87	46.87	46.87
RAG	47.61	47.61	47.61	47.61
BART	49.16	49.16	49.16	49.16
T5	49.29	49.29	49.29	49.29
BLINK	68.77	81.78	68.77	68.77
dev
BART + DPR	45.7	45.7	45.7	45.7
T5	46.58	46.58	46.58	46.58
RAG	46.7	46.7	46.7	46.7
BART	48.01	48.01	48.01	48.01
Table 10: WNED-CWEB
model	R-Precision	Recall@5	Accuracy	F1	KILT-AC	KILT-F1
test
BART	0.0	0.0	45.06	49.24	0.0	0.0
T5	0.0	0.0	43.56	50.61	0.0	0.0
BART + DPR	13.26	17.04	59.16	62.76	11.12	11.41
RAG	28.68	33.04	59.2	62.96	23.12	23.94
dev
BART	0.0	0.0	43.84	48.25	0.0	0.0
T5	0.0	0.0	47.24	51.73	0.0	0.0
BART + DPR	13.62	16.93	56.7	60.19	11.56	11.87
RAG	29.26	33.69	61.48	65.03	25.4	26.22
Table 11: T-REx
model	R-Precision	Recall@5	Accuracy	F1	KILT-AC	KILT-F1
test
BART	0.0	0.0	9.14	12.21	0.0	0.0
T5	0.0	0.0	9.02	13.52	0.0	0.0
BERT + DPR	40.11	40.11	6.93	37.28	4.47	27.09
BART + DPR	28.9	39.21	30.43	34.47	18.91	20.32
RAG	53.73	59.52	44.74	49.95	36.83	39.91
dev
BART	0.0	0.0	3.03	12.61	0.0	0.0
T5	0.0	0.0	1.58	10.8	0.0	0.0
BERT + DPR	45.6	58.49	34.96	44.79	29.08	32.85
RAG	65.36	73.07	47.42	57.98	42.64	48.35
Table 12: Zero Shot RE
model	R-Precision	Recall@5	EM	F1	KILT-EM	KILT-F1
test
BART	0.0	0.0	21.75	28.69	0.0	0.0
T5	0.0	0.0	19.6	27.73	0.0	0.0
BART + DPR	54.29	65.52	41.27	49.54	30.06	34.72
BERT + DPR	60.66	46.79	38.64	47.09	31.99	37.58
RAG	59.49	67.06	44.39	52.35	32.69	37.91
dev
BART	0.0	0.0	26.15	32.06	0.0	0.0
T5	0.0	0.0	25.2	31.88	0.0	0.0
BART + DPR	54.25	64.99	45.05	52.98	31.62	35.84
BERT + DPR	60.03	45.06	42.58	50.43	35.32	39.84
RAG	60.31	65.47	48.78	56.1	36.31	40.64
Table 13: Natural Questions
model	R-Precision	Recall@5	EM	F1	KILT-EM	KILT-F1
test
BART	0.0	0.0	15.37	21.97	0.0	0.0
T5	0.0	0.0	12.64	19.57	0.0	0.0
BERT + DPR	25.04	10.4	11.29	17.35	0.74	1.26
BART + DPR	25.04	10.4	25.18	34.07	1.96	2.53
RAG	30.59	12.59	26.97	36.03	3.21	4.1
dev
BART	0.0	0.0	16.86	23.81	0.0	0.0
T5	0.0	0.0	12.66	19.74	0.0	0.0
BERT + DPR	24.62	10.7	10.82	16.96	0.96	1.34
BART + DPR	24.62	10.7	25.75	35.2	1.96	2.46
RAG	30.76	12.29	27.68	37.37	3.14	3.87
Table 14: HotpotQA
model	R-Precision	Recall@5	EM	F1	KILT-EM	KILT-F1
test
BART	0.0	0.0	32.39	39.85	0.0	0.0
T5	0.0	0.0	18.11	27.83	0.0	0.0
BART + DPR	44.49	56.99	58.55	67.79	31.4	35.34
BERT + DPR	43.4	31.45	70.38	74.41	34.48	36.28
RAG	48.68	57.13	71.27	75.88	38.13	40.15
dev
BART	0.0	0.0	32.54	39.58	0.0	0.0
T5	0.0	0.0	25.79	33.72	0.0	0.0
BERT + DPR	40.87	29.96	70.24	74.21	32.9	34.48
BART + DPR	45.36	56.72	59.28	68.31	32.56	36.36
RAG	49.26	56.93	61.73	67.12	36.13	38.71
Table 15:TriviaQA
model	R-Precision	Recall@5	Rouge-L	F1	KILT-RL	KILT-F1
test
T5	0.0	0.0	19.08	16.1	0.0	0.0
BART	0.0	0.0	20.55	19.23	0.0	0.0
RAG	11.0	22.92	14.05	14.51	1.69	1.79
BART + DPR	10.67	26.92	17.41	17.88	1.9	2.01
dev
T5	0.0	0.0	21.02	18.36	0.0	0.0
BART	0.0	0.0	22.69	22.19	0.0	0.0
RAG	16.39	27.27	16.11	17.24	2.65	2.88
BART + DPR	16.32	21.11	18.53	18.75	2.87	2.89
Table 16: ELI5
model	R-Precision	Recall@5	Rouge-L	F1	KILT-RL	KILT-F1
test
BART	0.0	0.0	11.84	12.96	0.0	0.0
T5	0.0	0.0	12.58	13.49	0.0	0.0
TransMemNet	18.38	18.38	9.92	11.5	1.83	2.23
BART + DPR	25.48	55.1	13.56	15.55	3.88	4.52
RAG	57.78	74.63	11.83	13.22	8.04	9.1
dev
BART	0.0	0.0	12.05	13.35	0.0	0.0
T5	0.0	0.0	12.8	13.28	0.0	0.0
BART + DPR	0.0	0.0	13.23	15.03	0.0	0.0
RAG	46.73	66.61	12.03	13.42	7.01	7.69
Table 17: Wizard of Wikipedia
2 BART predictions:
4 v E0: 'Bucharest' -> https://en.wikipedia.org/wiki/Bucharest
5 x E1: 'Romania' (gold: 'Romania national football team')
6 x E2: 'Anghel lordanescu' (gold: 'Anghel lordanescu')
7 v E3: 'North Macedonia national football team'  $\rightarrow$  https://en.wikipedia.org/wiki/\~ North_Macedonia_national_football_team
8 x E4: '1998 FIFA World Cup' (gold: 'FIFA World Cup')
9 v E5: 'Viorel Ion' -> https://en.wikipedia.org/wiki/Viorel_Ion
10 v E6: 'Liviu Ciobotariu' -> https://en.wikipedia.org/wiki/Liviu_Ciobotariu
11 v E7: 'Europe' -> https://en.wikipedia.org/wiki/Europe
12 v E8: 'North Macedonia' -> https://en.wikipedia.org/wiki/North_Macedonia
13 v E9: 'UEFA Champions League' -> https://en.wikipedia.org/wiki/UEFA_Champions_League
14 v E10: 'Adrian Ilie'  $\rightarrow$  https://en.wikipedia.org/wiki/Adrian_Ilie
15 v E11: 'Galatasaray S.K. (football)' -> https://en.wikipedia.org/wiki/Galatasaray_S.K_. (football)
16 v E12: 'Bogdan Stelea' -> https://en.wikipedia.org/wiki/Bogdan_Stelea
17 v E13: 'Florin Prunea' -> https://en.wikipedia.org/wiki/Florin_Prunea
18 v E14: 'Dan Petrescu'  $\rightarrow$  https://en.wikipedia.org/wiki/Dan_Petrescu
19 v E15: 'Daniel Prodan'  $\rightarrow$  https://en.wikipedia.org/wiki/Daniel_Prodan
20 v E16: 'Liviu Ciobotariu' -> https://en.wikipedia.org/wiki/Liviu_Ciobotariu
21 v E17: 'Iulian Filipescu' -> https://en.wikipedia.org/wiki/Iulian Filipescu
22 v E18: 'Gheorghe Hagi'  $\rightarrow$  https://en.wikipedia.org/wiki/Gheorghe_Hagi
23 v E19: 'Gheorghe Popescu' -> https://en.wikipedia.org/wiki/Gheorghe_Popescu
24 x E20: 'Constantinos Galca' (gold: 'Constantin Galca')
25 v E21: 'Basarab Panduru' -> https://en.wikipedia.org/wiki/Basarab_Panduru
26 v E22: 'Dorinel Munteanu' -> https://en.wikipedia.org/wiki/Dorinel_Munteanu
27 v E23: 'Gheorghe Craioveanu'  $\rightarrow$  https://en.wikipedia.org/wiki/Gheorghe_Craioveanu
28 x E24: 'Ion Danciuulescu' (gold: 'Ionel Danciuulescu')
29 v E25: 'Viorel lon' -> https://en.wikipedia.org/wiki/Viorel_lon
31 F1-score = 87.52
32 KILT-F1-score = 21/26 = 80.77
33 EM  $= 21 / 26 = 80.77$
34 KILT-EM-score = 21/26 = 80.77
Figure 4: Entity linking BART predictions, schematic of 25 input-output pairs condensed, in each one a single entity in tagged.
(a) FEVER, dev data discarded  $26.03\%$  (3675), test data discarded  $27.7\%$  (3869).
(b) Natural Questions, dev data discarded  $16.12\%$  (595), test data discarded  $15.59\%$  (287).