Autoregressive Entity Retrieval
Authors: Nicola De Cao
Gautier Izacard
Sebastian Riedel
Fabio Petroni
Venue: ICLR 2021 Spotlight (2021)
Keywords: entity retrieval
document retrieval
autoregressive language model
entity linking
end-to-end entity linking
entity disambiguation
constrained beam search

Abstract: Entities are at the center of how we represent and aggregate knowledge. For instance, Encyclopedias such as Wikipedia are structured by entities (e.g., one per Wikipedia article). The ability to retrieve such entities given a query is fundamental for knowledge-intensive tasks such as entity linking and open-domain question answering. One way to understand current approaches is as classifiers among atomic labels, one for each entity. Their weight vectors are dense entity representations produced by encoding entity meta information such as their descriptions. This approach leads to several shortcomings: (i) context and entity affinity is mainly captured through a vector dot product, potentially missing fine-grained interactions between the two; (ii) a large memory footprint is needed to store dense representations when considering large entity sets; (iii) an appropriately hard set of negative data has to be subsampled at training time. In this work, we propose GENRE, the first system that retrieves entities by generating their unique names, left to right, token-by-token in an autoregressive fashion and conditioned on the context. This enables us to mitigate the aforementioned technical issues since: (i) the autoregressive formulation allows us to directly capture relations between context and entity name, effectively cross encoding both; (ii) the memory footprint is greatly reduced because the parameters of our encoder-decoder architecture scale with vocabulary size, not entity count; (iii) the exact softmax loss can be efficiently computed without the need to subsample negative data. We show the efficacy of the approach, experimenting with more than 20 datasets on entity disambiguation, end-to-end entity linking and document retrieval tasks, achieving new state-of-the-art or very competitive results while using a tiny fraction of the memory footprint of competing systems. Finally, we demonstrate that new entities can be added by simply specifying their unambiguous name. Code and pre-trained models at https://github.com/facebookresearch/GENRE.

Published as a conference paper at ICLR 2021
AUTOREGRESSIVE ENTITY RETRIEVAL
Nicola De Caolé; Gautier Iz. Sebastian Riedel Fabio Petroni
'University of Amsterdam, Facebook Al Research ENS, PSL University, IInii,, 'University College London nicola. d il. com, {gi sizaaaad,, sriedel, fabiopetroniJefb.com
ABSTRACT
Entities are at the center of how we represent and aggregate knowledge. For in- stance, Encyclopedias such Wikipedia structured by entities (e.g., one per Wikipedia article). The ability to retrieve such entities given query is fundamen- tal for knowledge-intensive tasks such as entity linking and open-domain question answering. One way to understand current approaches is as classifiers among atomic labels, one for each entity. Their weight vectors are dense entity repre- sentations produced by encoding entity meta information such as their descrip- tions. This approach leads several shortcomings: (i) context and entity affinity is mainly captured through vector dot product, potentially missing fine-grained interactions between the two; (ii) large memory footprint is needed to store dense representations when considering large entity sets; (iii) an appropriately hard set negative data has to be subsampled at training time. In this work, we propose GENRE, the first system that retrieves entities by generating their unique names, left to right, token-by by-koken in an autoregressive fashion and conditioned on the context. This enables to mitigate the aforementioned technical issues since: (i) the autoregressive formulation allows us to directly capture relations between context and entity name, effectively print is greatly reduced because the parameters of our encoder-decoder architec- ture scale with vocabulary size, not entity count; (iii) the exact softmax loss can be efficiently computed without the need to subsample negative data. We show the efficacy of the approach, experimenting with more than 20 datasets on entity disambiguation, end-to-end entity linking and document retrieval tasks, achieving new state-of-ther ofrt or very competitive results while using a tiny fraction of the memory footprint of competing systems. Finally, we demonstrate that new entities can be added by simply specifying their unambiguous name. Code and pre-trained models at atlatps:: /ttp://gttths wwt. http:///www/..eeeeeeeeeeeeaaaaaa /GENE GENRE
encoding both; (ii) the memory foot-
INTRODUCTION
The ability retrieve the correct entity from large Knowledge Bases (KBs) given textual input is a fundamental building block for several applications (Ferrucci) 2012 Slawski 2015) Yang et al.) 2018ap. Most commercial recommendation systems, for instance, include their pipelines compo- nents to detect and disambiguate entity mentions in open text, in order to isolate relevant concepts from non-meaningful data (Slawski] 2015 Yang et al., 2018a). Another example are chat-bots and question answering systems, that are often equipped with retrieval components to surface specific KB entries (e.g.. Wikipedia articles) to find knowledge for sustaining a conversation or answering a
question (Ferrucci 2012, Chen et al. 2017 Lewis et al. 2020b Roller et al. 2020).
Although there has been extensive previous work on entity retrieval (e.g. Hoffart et 2011 Pic- cinno & Ferragina 2014 Huang et al. 2015 Te & Titov 2018 |018] Logeswaran et al. 2019 Broscheit, 2019 Wu et al., 2020, to name just few) there is a common design choice to most current so- lutions: entities are associated with unique atomic label and the retrieval problem can be in- terpreted as multi-class classification across these labels. The match between input and label is calculated through bi-encoder (Wu et al.. 2020 Karpukhin et al.] (020): dot product between dense vector encodings of the input and the entity's meta information (such as title and description).
Work done during internship with Facchook AT Research.

---

Published as a conference paper at ICLR 2021
Superman saved
From 1905 to 1985 Owhango had [START] railway station [END)
[START] Farnese Palace [END] one the most important palaces in the city Rome
[START] Metropolis [END)
1 Metropolis (comics)
Owhango railway station
Palazzo Farnese
Metropolis (1927 film)
Train station
Palazzo dei Normanni
3 Metropolis- Hastopppli--Htttngggg algorithm
Owhango
3 Palazzo della Farnesina
(a) Type specification.
(b) Composing from context.
(c) Trans
Which US nuclear reactor had major accident 1979?
Stripes had Conrad Dunn featured
What is the espital of Holland?
1 Netherlands
Three Mile Island accident
Conrad Dunn
Capital the Netherlands
2 Nuclear reactor
2 Stripes (film)
3 Holland
Cherooyl disaster
3 Kris Kristofferson
(d) Entity normalization.
(c) Implicit factual knowledge.
(f) Exact copy.
Figure I: Examples of entities correctly retrieved from GENRE (we show only the top-3 rank). On the top three entity disambiguation instances and on the bottom three document retrieval instances, two for open-domain question answering and one for fact checking. All of them are cast as sequence- to-sequence problems while inference is done using constrained beam search. Gold entities in bold. Sub-captions indicate the type of interaction between the input context and the entity names required.
Critically, this formulation enables sub-linear search using modern carnuuu--inne--rroddee libraries (Johnson et al.) 2019) and hence supports retrieving from large entity databases.
Unfortunately, the classifier approach to entity retrieval also has several shortcomings. First, unless a costly cross-encoder is used for re-ranking (Wu er al., (020), the dot-product can miss fine-grained interactions between input and entity meta information (Humeau et al. 2020). Second, storing dense vectors for the whole KB requires a large memory footprint, especially in real-world scenarios (i.e., 224B store (024-dimensional vectors for all of the ~6M Wikipedia pages), and the size linearly grows with the addition of new entities. Third, computing an exact softmax over all entities is very expensive, hence current solutions need to subsample negative data (Logeswaran et al., 2019] Karpukhin et al.) (2220) at training time. Tuning appropriately hard set of negative instances can be challenging and time-consuming. Finally, existing systems can suffer from a cold-start problem since they cannot represent entities about which they have not yet gathered sufficient information, in the form, for instance, of a textual description or a set of relations with the cxisting entities.
The treatment of entity identifiers as atomic labels in classifier ignores the fact that we often have unambiguous, highly structured and compositional entity names. Wikipedia, for instance, associates unique titles to articles | that may be the name of the subject or description of its topic, well as potential distinctive information to disambiguate (see Figure 1 for some examples). These entity names often interact with mention contexts in predictable and regular fashion. For example, often entity names are identical with the mention strings that refer to them (e.g.. Fig. Tr.. When this is not possible, they might be composed of tokens in the context (c.g.. Fig. Ib). include a type specification that can inferred (c.g., Fig. Iay. be the translation of the string mention (c.g., Fig, [Ic). require 'normalization' such as referring to the correct alias of a mention (c.g.. Fig. Id] or require factual knowledge that might be stored in the parameters of a model (e.g.. Fig. Tep. These observations suggest that inputs could be translated into unique entity names, word by word, instead of being classified among a huge set of options.
In this paper, we propose GENRE (for Generative ENtity REtrieval), the first entity retriever that exploits requence-to-sequeee conditioned on the context. Concretely, GENRE uses a wransformer-base architecture, pre-trained with a language modeling objective (i.e., we use BART weights from Lewis et al. (2020a)) and fine-tuned to generate entity names. This architecture has been shown to retain factual knowledge
architecture to generate entity names in an autoregressive fashion
We use entity name to refer to the corresponding Wikipedia article title throughout the rest of the paper. oftee in the form of description in parentheses after the name. Wikipedia naming conventions are de- scribed in wtthttss://// //en. wt pedi a oott
2

---

Published as a conference paper at ICLR 2021
to extent (Petroni al. 2019) and language translation skills (Radford et al. 2019) among other things, both desirable properties for an entity retriever. Naturally, the generated output might not always be valid entity name. To solve this problem, GENRE employs constrained decoding strategy that forces each generated name to be in a predefined candidate set.
The autoregressive formulation allows to directly capture the aforementioned relations between context and entity name, effectively cross encoding both. Also, the memory footprint required is orders of magnitude smaller than current systems, since the parameters of segueence-t-sequemne model scale linearly with the vocabulary size, not entity count. Moreover, the exact softmax can be computed efficiently for each output token (i.e., all non-gold tokens are considered negative), thereby eliminating the need for negative data downsampling. Finally, our model never accesses any explicit meta-information about the entity beyond their title, hence new entities can be added by simply appending their unambiguous name to the candidate set (e.g.. Fig. Tb refers to entity added after training).
We empirically evaluate the performance of GENRE on more than 20 datasets, spanning three families of tasks: (i) entity disambiguation, using popular datasets and settings (both in and out-of- domain); (ii) end-to-end entity linking, with the GERBIL benchmarking tool (Röder et al. 2018)) by using a novel dynamically markup-constrained decoding strategy; (iii) document retrieval, with the recently proposed KILT benchmark (Petroni et al. 2020b which spans 5 different sub-tasks. Our models achieve state-of-theeart or very competitive results on nearly all datasets, often with sub- stantial improvement (+13.7 precision points KILT for retrieval on average). Further, we show that compared with recent models, GENRE requires substantially less memory (~~20 times smaller footprint on average). Finally, we demonstrate that our model can be applied in scenarios where the only entity information available is its name.
We organize the paper as follows: in Section we describe our problem formulation. Then, in Section 3 we present GENRE and eventually in Section&Jwe extensively evaluate our method on the aforementioned settings. We will release code and pre-processed data reproduce our experiments.
2 ENTITY RETRIEVAL
We assume to have a collection of entities E (e.g.. Wikipedia articles) where each entity is an entry in a Knowledge Base (KB) such as Wikipedia. We want to approach the following retrieval problem: given a textual input source x (e.g.. question), model has to return the most relevant entities from E with respect to x. We assume that each E & is uniquely assigned to a textual representation (i.e., its name): a sequence tokens (e.g., Wikipedia pages are identified by their titles).
A particular instance of this problem is Entity Disambiguation (ED) (see Figure D for an example) where an input x is annotated with a mention and a system has to select either its corresponding entity from ε, to predict that there is no corresponding entry in the KB. Another instance is page- level Document Retrieval (DR) where the input x is intended as a query and E as a collection of documents identified by their unique titles (e.g., Wikipedia articles).
3 METHOD
We address the retrieval problem with sequence-to-sequene model that generates textual en- tity identifiers (i.e., entity names). Concretely, GENRE ranks each e E E by computing a score with an autoregressive formulation: score(e|æ) = (o()xx) N (o(sslue).z) where is the set of N tokens in the identifier of e, and 0 the parameters of the model. We take advantage of fine-tuning the BART (Lewis et al.] 2020a) pre-trained language model. We train GENRE using a standard seq2seq objective, i.c., maximizing the output sequence likelihood with teacher fore- ing (Sutskever et al. [011 2014) and regularized with dropout (Srivastava et al. 2014) and label smoothing (Szegedy et al., 2016). Concretely, we use the objective that is typically used for neural machine translation (NMT, Wu al. 2016), that is maximizing po with respect to model's parameters which, due to the factorized formulation, can be calculated exactly. We do not need negative sampling to approximate the loss normalizer.
3

---

Published as a conference paper at ICLR 2021
da Vinci DiCaprio (TV series)
Levnardo
In 1503,-.
In 1503, [Leonardo
In 1503, [I.onardo]( (connadd
(a) Outside: we can either continue to generate the in- put or start new mention.
(b) Inside mention: we either contimue to generate the input or end the current mention.
(c) Inside entity link: can either gen- erate from the entities prefix trie or close the generated pretix is a valid entity.
Figure 2: Example of dynamically constrained Markup decoding for entity linking using "/n 1503, Leonardo began painting the Mona Lisa." as input. There are 3 cases: when we are outside a men- tion/entity (a), inside mention generation step (b), and inside an entity link generation step (c). The model is supposed to output the input source annotating mentions and pointing them to the respective entities: "In 1503, [Leonardo]] Leonardo da Vinci) began painting the [Mona Misa)(Mona Mona Lisa)".
3.1 INFERENCE WITH CONSTRAINED BEAM SEARCH
Naturally, test time, we could compute score for every element in & and then sort them. Un- fortunately, this might be prohibitively expensive when & is very large (e.g.. Wikipedia has ~6M entities). Hence, we exploit Beam Search (BS, Sutskeerr et al., 2014), an established approximate decoding strategies to efficiently navigate the search space. Instead of explicitly scoring all entities in E, we search for the top-k entities that using BS implies that the time cost of our retriever does not depend on the size of &, but only on the size of the beams and the average length of entity representations as we do autoregressive generation. The average length of entity representations is tractable (e.g., Wikipedia titles have 6 BPE tokens on average) and we follow standard NMT settings where i is small (e.g., 10).
decoding from our model using BS with & beams. Note
Since want output only entities from & we cannot traditional BS while decoding. Indeed, allowing to generate any token from the vocabulary at every decoding step might lead the model to generate output strings that are not valid identifiers. Hence, we resort Constrained BS, forcing to only decode valid entity identifiers. BS only considers one step ahead during decoding so we can only constrain the generation of a single next token conditioned on the previous ones. Thus, we define our constrain in terms of a prefix tree T (aka trie) (Cormen et al. (209) where nodes are annotated with tokens from the vocabulary. For each node & T. its children indicate all the allowed continuations from the prefix defined traversing the trie from the root to t.
See Figure in Appendix C for an exampled of trie. When the number of allowed outputs is tractable (e.g., generating Wikipedia title among ~6M) the trie is relatively small it can be pre- computed and stored into memory (e.g., constraining on Wikipedia titles using the BART tokenizer produces trie with ~-6 leaves, ~1MM internal nodes that occupied ~60MMB of disk space). We employed the constraints masking the log- peg-rbbabilities of the invalid tokens and not their logits (i.c., we do not re-normalize the probability over the vocabulary)
3.2 AUTOREGRESSIVE END-TO-END ENTITY LINKING
We additionally extend our autoregressive framework address end-to-end Entity Linking (EL) where, given a document, a system has to both detect entity mentions and link those mentions to their respective KB entities. In this setting, we train the model to predict the source input again but with annotated spans. We use a Markup annotation where spans boundaries are flagged with special tokens and accompanied by their corresponding entity identifiers.
Differently from setting where the output space is relatively small (c.g., a pre-defined set E), the space of annotated outputs is exponentially large. Hence, it is intractable to pre-compute a trie for decoding, and compute it dynamically instead. In Figure
show example. At each generation step, the decoder is either generating a mention span, generating a link to a mention, continuing from the input source. When outside mention/entity step the decoder has only two options: (i) to continue by copying the next token from the input source, or (ii) to generate the start of mention token (i.e., '(') which makes the decoder enter the mention generating phase. While
We experimented with hoth versions and find masking the log-prohability more effective.
4

---

Published as a conference paper at ICLR 2021
generating mention, the decoder has either to continue with the next token in the input source or to generate the end of mention token (i.e.. 'I') which makes the decoder enter the entity generating phase. Finally, when generating an entity, the decoder employs the entities trie such that it can only output a valid entity identifier as in Constrained Beam Search explained above.
4 EXPERIMENTS
We extensively evaluate GENRE on more than 20 datasets across 3 tasks: Entity Disambiguation, end-to-end Entity Linking (EL), and page-level Document Retrieval. We describe the experimental settings in Section 4.1 where we discuss results in Section 4.2 All experiments are in English.
4.1 SETTINGS
Entity Disambiguation (ED) We reproduce the setting of Le & Titov (018) using the same candidate sets, in-domain and out-of-domain datasets, and evaluating using the InKB micro-Fj. We train GENRE feeding each document where single mention is flagged with two special start and end tokens and the target output is the textual representation of the corresponding entity. At test time, we decode using constrained beam search with a trie obtained using the provided candidate set (i.e.. a subset of E). As large generative models benefit from large amount of data, we first pre-train GENRE on the BLINK data (Wu et al. 2020). i.e., 9M unique triples document-menntonn-nnle Wikipedia. Then, for the in-domain scenario, we fine-tune using the AIDA-CONIL dataset (Hoffart ct al. 2011) For the out-of-domain scenario, we evaluate on five test sets: MSNBC, AQUAINT, ACE2004, WNED-CWEB (CWEB) and WNED- WNED-WIKI (WIKI) (Gabrilovich et al. 2013 Guo &
from
Barbosa 2018). More task details and hyperparameters setting are reported Appendix A.1
End-to-End Entity Linking (EL) For EL, we reproduce the setting of Kolitsas et al. (2018) us- ing the same in-domain and out-of-domain datasets as well as evaluating the InKB micro-F, F1 on the GERBIL benchmark platform (Röder et al. 2018). Similarly to the ED setting, we first pre-traine model on all abstract sections from Wikipedia jenrihedd by string matching heuristic solve einrieedieerriiheee
co-references (i.e., if there is a string that matches exactly with another hyperlink we also add it to the dataset mention/entity pairs). Then, for the in-domain scenario, we fine-tune using the AIDA- CoNLL dataset. We evaluate on seven out-of-domain test sets: MSNBC, Derczynski (Der) (Der-) czynski et al., 2 KORE 50 (K50) (Hoffart et al., 2012), N3-Reuters- 128 (R128). N3-RSS-500 (R500) (Röder et al. 2014), and OKE challenge 2015 and 2016 (OKE15 and OKE16) (Nuzzolesc) ct al. 2015). More task details and hyperparameters setting are reported in Appendix A.2
Page-level Document Retrieval (DR) For this setting, we test GENRE on all the KILT bench- mark tasks (Petroni et al. 2020bb. Here, whole Wikipedia is used as the candidate set and we evalu- ate using R-precision (Beitzel et al. 2009). KILT consists of five tasks that use the same Wikipedia dump as knowledge source: fact checking with FEVER (Thorne et al.) 2018); open domain ques- tion answering using Natural Questions (Kwiatkowski et al. 2019), HotpotQA (Yang et al., 20185), TriviaQA (Joshi al. (017)) EL15 (Fan al. 2019); slot filling with T-REx (Blsahar al. 2018). Zero Shot RE (Levy et al. 2017); entity disambiguation on AIDA CONLL YAGO, WIKD-WIKI
and WNED-CWEB; dialogue with Wizard of Wikipedia (Dinan et al., 2019). We train GENRE on BLINK and all KILT data simultancously with single model More details on the hyperparameter setting are reported in Appendix A.3
4.2 RESULTS
Overall, GENRE achieves very competitive results in all of the three settings being the best per- forming system on average across all of them. See Appendix[for examples of inputs, ground truth and model predictions for all of the three tasks. In the following, we discuss how GENRE com- pares to SOTA systems as well as showing some quantitative analysis on its memory footprint, how
'It is based on the 2019/08/01 Wikipedia dump pre-pracessed by Petroni et al. 2020b).
'Not that not all dataset available in KILT have training set. Concretely, we train FEVER, Natural Questions, HorpotQA, TriviaQA, T-REx, Zero Shot RE, AIDA CoNL.L-YAGO, and Wizard of Wikipedia.
5

---

Published as a conference paper at ICLR 2021
In-domain AIDA
Out-of-domain
Method
MSNBC
AQUAINT
ACE2004
CWER
WIKT
Avg.
Ganea & Hofmann 2017
92.2
93.7
88.5
88.5
77.9
77.5
86.4
Guo & Barbosa 2018
89
92
87
88
77
84.5
86.2
Yang et al.
20
95.9
92.6
89,9
88.5
81.8
79.2
88.0
Shahbazi et al.
2019
93.5
92.3
90.1
88.7
78.4
79.8
87.1
Yang et al.
201
93.7
93.8
88.2
90.1
75.6
78.8
86.7
Le 1itov
2019
89.6
78.2
81.7
Fang et al.
2019
94.3
92.8
87.5
91.2
78.5
82.8
87.9
BLINK w/o candidate se
79.6
80.0
80.3
82.5
64.2
75.5
77.0
GENRE
93.3
94.3
89.9
90.1
77.3
87.4
88.8
Ablations
GENRE only AIDA data
88.6
88.1
77.1
82.3
71.9
71.7
80.0
GENRE only BLINK data
89.3
93.3
90.9
91.1
76.0
87.9
88.1
GENRE w/o candidate set
91.2
86.9
87.2
87.5
71.1
86.4
85.1
GENRE w/o constraints
86.4
80.0
81.7
82.1
66.0
81.1
79.6
Table 1: Micro Fi (InKB) on the in-domain test set and five out-of-domain test sets for the named entity disambiguation task. Bold indicates best model and underline indicates second best (not for ablations). *WIK is usually considered out-of-domain but note that all methods use part of Wikipedia train. **results taken from https: htts:://gtthbbt com http://///ccbbbooeeeec.ccc BLINK and normalized to accommodate entities not in KB.
it exploits the structured the entity name space, and how it behaves on a cold-start scenario where new unseen entities added to the KB (descriptions of those entities unobserved).
Comparing GENRE to SOTA systems ED the difference in average Fi score between GENRE and the second best performing system is small (i.e., +0.8) however, ED is an established task with more than decade of research that benchmarked on those datasets. Indeed all systems reported in Table achieved high and similar results even if they were taken from three years back.
The improvements on EL are instead more evident. GENRE is the best in-domain system for AIDA while performing remarkably well also on the out-of-domait setting (e.g., +13 F1 points on Derezynski, and +4.7 on KORE50). Noticeably, in two datasets (OKE15 and OKE16) our model performs poorly. However, these datasets are annotated with coreference (pronouns and common nouns are linked to entities) while our model was not specifically trained for that. Conversely, most of the other systems, have mention detection component in their pipelines that be trained or biased to also solve these cases. We considered out of the aim of this work to additional train and evaluate coreference and we leave it for future work.
On page-level DR, the superiority of GENRE is remarkable. Our model is the best performing system across all 5 KILT tasks and all datasets except Natural Questions where it is the sec- ond best. We achieve + +13.7 R-precision points on average with respect to the best performing baseline. In Table we compare GENRE against all methods reported in the public leaderboard: DPR (Karpukhin et al. 2020), DPR+BERT (Devlin et al.) 2019). DPR+BART if-idf (Leskoved al. 2014). RAG (Lewis et al. 2020b). and BLINK+flair (Wu et al. 2020 Akbik et al. 2019). No model except ours was trained on the entire KILT dataset at the same time. A RAG model was trained for every single task as well as for DPR+BERT. Note that this gives and advantage RAG and DPR+BERT to specialize
single tasks where we have only a single model to solve all of them which still performs better. We speculate that multi-task training could have helped since the all tasks share common objective to retrieve entities. Both DPR and BLINK were not trained specifically on KILT. However, DPR was trained using several QA datasets which include Natural Question and TriviaQA. In Appendix B report additional results where do not pre-train or fine-tune our models for both the ED and retrieval setting in Table and & respectively. When we train GENRE only in the DPR BLINK data, model still outperforms them.

---

Published as a conference paper at ICLR 2021
In-domain AIDA
Out-of-domain
Method
MSNBC
Der
K50
R128
R500
OKE15*
OKE16
Avg.
Hoffart et al.
2011 1
72.8
65.1
32.6
55.4
46.4
42.4
63.1
0.0
47.2
Stemmeetz & Sack
2013
42.3
30.9
26.5
46.8
18.1
20.5
46.2
46.4
34.7
Moro ct al. 2014
48.5
39.7
29.8
55.9
23.0
29.1
41.9
37.7
38.2
Kolitsas et al. 2018
82.4
72.4
34.1
35.2
50.3
38.2
61.9
52.7
53.4
Broscheil 2019
79.3
Martins ct al. 2019
81.9
van Hulst ct al.
2020
80.5
72.4
41.1
50.7
49.9
35.0
63.1
58.3
56.4
GENRE
83.7
73.7
54.1
60.7
46.7
40.3
56.1
50.0
58.2
Table 2: Micro Fi (InKB) on the in-domain test set and four out-of-domain test sets for the entity linking task. Bold indicates best model and underline indicates second best. *annotated with coref- erence (note that we do not train/evaluate our model to link pronouns and common nouns). fresults from the Wikipedia 2019 setting as opposed to the 2014 setting (older dump and fewer entities).
Memory Footprint GENRE is not only performing better than other SOTA models on DR but it has significant reduction of memory footprint (disk space). In Figure we compare the number of model/index parameter against DPR, RAG, and BLINK. GENRE uses an order of magnitude less parameters (millions instead of billions) store the entity index because it just has to use prefix tree of the entity names as opposed to dense vector for each entity. Concretely, GENRE occupied 14 times less memory than BLINK and 34 times less memory than DPR.
Exploiting the Structured Name Space We investigated some properties of GENRE, comparing two variants of our model to BLINK on the ED task (using WNED-KILT validation set): one trained to generate entity names and another to generate numerical identifiers (IDs). All models are trained on the same data and we report results in Figure When there is an exact match between a mention and its entity name, both BLINK and GENRE almost always make an accurate prediction. Different is the case of partial and no match: GENRE performance is much higher suggesting that our model uses the context more effectively, as the autoregressive formulation allows to cross-encode mention context and entity candidates directly capturing fine-grained interactions between the two. More- over, when we switch to predicting IDs, the performance drops drastically (-20.3 points on average) indicating that it is important that entity names are meaningful, structured and compositional (as they are in Wikipedia) conversely to atomic IDs. Surprisingly, when there is no overlap between a mention-entity pair, performance are still relatively high by using IDs. This suggests that the model is good at memorizing and recalling identifiers even if numeric.
Ablation study We here discuss an ablation study on the entity disambiguation task (see Table I. Due to space limitation, we discuss an ablation study on document retrieval in Appendix B.2 In Table [ GENRE only AIDA or BLINK data indicates the ablation for which we only train on one of the two datasets (i.e., only fine-tuning) GENRE (full) is also used with constrained decoding (see Section 3 and in combination with a candidate set (as provided by Lye & Titov 2018). GENRE without candidate set denotes ablating the provided (and small) candidate set and therefore using all the entities in the KB (in our case Wikipedia) as candidates. GENRE without constraints in- dicates ablating constrained decoding which implies no of the provided candidates set but also unconstrained generation (i.e., the model may generate entity names that are not in the KB). Even- tually, using constrained generation and exploiting the candidate sets proved useful. Training only on AIDA data is insufficient to get high Fi (but AIDA is quite small compared to the 9M datapoints of BLINK data).
Entity frequency The performance of a model naturally depends on how many times entities ap- pear in the training data. We show the data distribution of the mention-entity frequency in Figure Most of the pairs appears in Wikipedia (10931 13354) where 2423 do not (first bin). The aver- age accuracy is 82.5% but noticeable it is higher for mention-entity pairs that are more frequent (right side of the plot). The accuracy for pairs that do not appear in Wikipedia is substantially lower than the average suggesting that those are harder cases (the very end tail of the distribution). The degradation in performance is minimal indicating that our model is good at predicting rare entities.
7

---

Published as a conference paper at ICLR 2021
Fact Check.
Entity Disambiguation
Slot Filling T-REx
Open Domain QA
Dial.
Model
FEV
AY2
WnWi
WnCw
zsRE
NQ
HoPo
TQA
ELK
WoW
Avg.
DPR BERT
72.9
40.1
60.7
25.0
43,4
DPR
55.3
1.8
0.3
0.5
13.3
28.9
54.3
25.0
44.5
10.7
25.5
23.6
If-idr
50.9
3.7
0.24
2.1
44.7
60.8
28.1
34.1
46.4
13.7
49.0
30.5
DPR BART
55.3
75.5
45.2
46.9
13.3
28.9
54.3
25.0
44.4
10.7
25.4
38.6
RAG
61.9
72.6
48.1
47.6
28.7
53.7
59.5
30.6
48.7
57.8
47.3
BLINK flair
81.5
80.2
68.8
59.6
78.8
24.5
46.1
65.6
38.2
56.0
GENRE
83.6
89.9
87.4
71.2
79.4
95.8
60.3
51.3
69.2
15.8
62.9
69.7
Table 3: R-Precision for page-level retrieval KILT test data. Bold indicates the best model and underline indicates the second best. For our model, we indicated what datasets we used for training.
Model
Memory
Param.
Index
Type (omportt)
BLINK
GENRE
IDs*
DPR
70.9GB
220M
15B
Exact match (1543)
97.8
96.6
76.0
RAG
40.4GB
626M
15B
Partial match (133)
70.7
86.9
63.8
BLINK
30.1GB
680M
6B
No match (322)
49.4
59.9
55.0
GENRE
2.1GB
406M
17M
Total (3996)
81.0
88.8
68.5
Table Comparison between retrieval mod- els on memory (disk space) footprint and number model/index parameters.
Table 5: Different types of matches between men- tions and their entity names on the WNED-KILL *indicates GENRE trained on numerical identifiers.
Cold-start We manually collect 50 Wikipedia articles that were created in 202 simulate cold-start setting where new entities are added to the KB and the only entity information available their names. To create ED instances we resort hyperlinks pointing those entities other Wikipedia articles. 19 out of 50 mentions have an exact match with their respective entity names and them were correctly classified by GENRE. In combination with the results from Table 5 can conclude that GENRE has bias exactly copying the mention, and this helps unseen data. GENRE also correctly classified 14/31 of the remaining mentions (45.2%). This demonstrates the ability of our solution be applied in scenarios where entity metadata is unavailable (apart his name), a setting where, to the best of our knowledge, no existing system is capable to operate.
We additionally test how GENRE performs unseen mention-entity pairs WikilinksNED Cinseen-Mention data (Onoe & Durrett 2020) and we report all results in Table Appendix B. Surprisingly, GENRE performs almost the same for seen and unseen entity pairs (64.4 vs 63.2 ac- curacy) However, in the Onoe & Durrett (2020) setting we cannot guarantee entity descriptions have not been seen by BART during pre-training (given his training data contains Wikipedia).
RELATED WORKS
Casting NLP tasks with a structured input or output into sequence-to-sequencc problems has been explored for different problems, including semantic parsing (Rongali et al.) 2020), semantic role labelling (Daza Frank 2018), discourse representation structure parsing (Liu et al. 2018), gen- eration of fluent natural language responses from structured semantic representations (Balakrishnan ctt al. 2019). generation and parsing of abstract meaning representation (Konstas et al. 2017). In these works structured representation, of symbols compatible with seq2seq architecture. To the best our knowledge, we are the first cast entity retrieval sequence-to-sequence problem while decoding with an autoregressive formulation during inference.
graph for instance, is linearized into sequence
Related constrained generation mechanism, Daza & Frank (2018): Rongali al. (2020)
a copying mechanism in order to limit lexical deviations between the input and output strings. In these tasks, as well as for our problem, is natural promote a copying mechanism due to the input and the output proximity. A different type of constraint, a structural constraint, is used in Bal-
Note that both pre-training and fine-tuning dumps from 2019.
8

---

Published as a conference paper at ICLR 2021
disttbution
0.8
5.e=03
077
48c108
Accund
04
24c102
21e102
14e403
No
21
24
2
2
2
00e000
Mention-entily pair appearances
Figure 3: Accuracy per mention-entity pair frequency (in Wikipedia) on the validation sets of all Entity Disambiguation tasks in KILT.
akrishnan et al. (019)) maintain valid tree structure. Our constrained beam search encompasses both aspects, a copying mechanism that restrains the vocabulary and a structural constraint to ob- tain well-formed annotated output. In addition to these tasks with close input and output, the integration mechanism to guide the output of neural networks has been explored in various set- tings. Lexically constrained decoding has been used to force the inclusion of pre-specified words for machine translation (Hokamp & Liu 2017 Post & Vilar 2018), and image captioning (Anderson al. 2017). To the best of our knowledge, entity disambiguation, end-to-end entity linking, and query- query-based entity retrieval.
the first exploit constrained generation for
Nogueira al. (2020) propose to use eequence-to-seeeeee sequence model to re-rank document. Given query and a document the model is trained to output the words "true" or "false" depending on whether the document relevant or not. Differently from our approach for entity retrieval, it re- quires limited list of candidates documents, obtained with BM25 for instance, in order to be com- putationally possible. Massarelli al. (2019); Petroni et al. (2020a) explore the idea of using an autoregressive language model as neural retriever, by exploiting the implicit knowledge stored in their parameters to generate relevant sentences given a query. While intriguing, such solutions still lag behind retrievers with explicit knowledge access (e.g., explicit Wikipedia index). The idea of using a generative model for entity disambiguation was proposed in Petroni et al.] (2020b) they trained both BART and T5 in seq2seq fashion all KILT tasks (including ED). We panded that intuition generalizing on multiple tasks (end-to-end EL and page-level retrieval) as well as introducing constrained decoding for an efficient and effective search.
CONCLUSIONS
this work, we propose GENRE, a novel paradigm to addresses entity retrieval: generate entity names autoregressively. Entity names have several properties that might help (even humans) trieving them, including a compositional structure and a predictable interaction with the context. The autoregressive formulation allows to directly capture some of these properties, leading to several advantages with respect to current solutions, including an efficient way to cross encode men- tion context and entity candidates, much smaller memory footprint, and the ability to compute an exact softmax without the need to subsample negative data. We empirically show that these characteristics, combined with constrained decoding strategies, led state-of-the-art performance on a plethora of entity retrieval datasets, spanning entity disambiguation, end-to-end entity linking, and page-level document retrieval, while resulting in systems with remarkably contained memory footprint, a space reduction by factor of twenty on average. We additionally demonstrate that new entities can effectively considered in our system by simply appending their unambiguous name to the candidate set.
ACKNOWLEDGMENTS
Authors thank Patrick Lewis, Aleksandra Piktus, Michael Schlichtkrull, Ivan Titov, Jean Maillard, Edouard Grave, Sergio De Cao, Luisa Quarta for helpful discussions and technical support.

---

Published as a conference paper at ICLR 2021
REFERENCES
Alan Akbik, Tanja Bergmann, Duncan Blythe, Kashif Rasul, Stefan Schweter, and Roland Vollgraf. FLAIR: An easy-to-use framework for state-of-the-ar NLP. In Proceedings of the 2019 Confer- ence of the North American Chapter of the Association for Computational Linguistics (Demon- strations), pp. 54-59, Minneapolis, Minnesota, June 2019, Association for Computational Lin- guistics. doi: 10.18633/1//N19-40000 URL https: /wwwwwwww.....o ac www.a//wwww.....a.oooo..oo...o
N19-4010]
Peter Anderson, Basura Fernando, Mark Johnson, and Stephen Gould. Guided open vocabu- lary image captioning with constrained beam search. In Proceedings of the 2017 Conference on Empirical Methods Natural Language Processing, pp. 936-945, Copenhagen, Denmark, September 2017. Association for Computational Linguistics. doi: (0.18653/11/D7--1000 URL
https: www www.wclwwbbbbc.... orq/ wtt.:///aaaaaaaaaa/o1/o11100
Anusha Balakrishnan, Jinfeng Rao, Kartikeya Upasani, Michael White, and Rajen Subba. Con- strained decoding for neural NLG from compositional representations in task-oriented dialogue. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguis- fics, pp. 831-844, Florence, Italy, July 2019. Association for Computational Linguistics. doi: 10.18653/v1/P13 1tt////91-000010 1080. URL httpt::////.. wwwpwwaclleee.. http::/wwwwww.....orraaa.oo.oo//
Steven M. Beitzel, Eric C. Jensen, and Ophir Frieder. Average R-Precision, pp. 195-195, Springer US, Boston, MA, 2009. ISBN 978-0-3887399940-9 doi: 10.
URL
https://doo www.oo.g.......... 1007/978 08700-3877777000 387-39940
9_4
Samuel Broscheit. Investigating entity knowledge in BERT with simple neural end-to-end entity linking. In Proceedings of the 23rd Conference on Computational Natural Language Learning (CoNLL), pp. 677-683 Hong Kong, China, November 2019. Association for Computational Lin- guistics. doi: 10.18653/v1/K199 1063. URL https: //wp::///wwww. /www. aclweb. org/anthology. K19 1063
Danqi Chen, Adam Fisch, Jason Weston, and Antoine Bordes. Reading Wikipedia answer open-domain questions. Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 1870-1879, Vancouver, Canada, July 2017. Association for Computational Linguistics. doi: 10. 1/155//11/11--11111 URL https: /www. aclweb. org/ anthology/E17-111
Shuang Chen, Jinpeng Wang. Feng Jiang, and Chin-Yev Lin. Improving entity linking by modeling latent entity type information. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 34, pp. 7529- 75377557, 2020.
Thomas H Cormen, Charles E Leiserson, Ronald L. Rivest, and Clifford Stein. Introduction to algorithms. MIT press, 2009.
Angel Daza and Anette Frank. A sequence-to-sequenee model for semantic role labeling. In Pro- ceedings of The Third Workshop on Representation Learning for NLP, pp. 207-216, Melbourne, Australia. July 2018. Association for Computational Linguistics. doi: 10. 1/W 18-3027.
URL https: //www www wttp:/wwwwww....///////// acwweb. wwwp wwt.oo/gg/nnoog//.o.oooo 3027
Leon Derezynski, Diana Maynard, Giuseppe Rizzo, Marieke Van Erp. Genevieve Gorrell, Raphaël Troncy, Johann Petrak, and Kalina Bontcheva. Analysis of named entity recognition and linking for tweets. Information Processing & Management, 5 (2):32 49, 2015.
Jacob Devlin, Ming- Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chupter of the Association for Computational Linguistics: Human Language Technologies. Volume 1 (Long and Short Papers), pp. 4171-4186, Minncapolis, Minnesota, June 2019. Association for Computational Linguistics. doi: 10. 10.18653/1/N11--4422, URL https:
www. acwweb org/ anthologyy haoeollgyy/99911 1423
Emily Dinan, Stephen Roller, Kurt Shuster, Angela Fan, Michael Auli, and Jason Weston. Wiz- ard of wikipedia: Knowledge-pond poweeed conversational agents. Proceedings of the International Conference Learning Representations (ICLR), 2019.
10

---

Published as a conference paper at ICLR 2021
Hady Elsahar, Pavlos Vougiouklis, Arslen Remaci, Christophe Gravier, Jonathon Hare, Elena Sim- perl, and Frederique Laforest. T-rex: A large scale alignment of natural language with knowledge base triples. LREC, 2018.
Angela Fan, Yacine Jernite, Ethan Perez, David Grangier, Jason Weston, and Michael Auli. ELI5: long form question answering. In Anna Korhonen, David R. Traum, and Lluis Marquez (eds.), Proceedings of the 57th Conference of the Association for Computational Linguistics, ACL 2019, Florence, Italy, July 28- August 2, 2019, Volume 1: Long Papers, Pp. for Computational Linguistics. 2019. doi: 10.18653/v1/p19-1133 URL http:t::: / .org/ 10. 10.18653/v1/p19-1333
Association
Zheng Fang, Yanan Cao, Qian Li, Dongjie Zhang. Zhenyu Zhang, and Yanbing Liu. Joint entity linking with deep reinforcement learning. The World Wide Web Conference, pp. 438 447. ACM, 2019.
David A Ferrucci. Introduction to "this is watson". IBM Journal of Research and Development, 56 2012
Evgeniy Gabrilovich, Michael Ringgaard, and Amarnag Subramanya. Facel: Freebase annotation of clueweb corpora, version (release date 2013-06-26, format version 1, correction level 0).
Note: http://lemurrrojec.. org/Clueweebb////CC//CC
by, 5:140, 2013.
Octavian-Eugen Ganea and Thomas Hofmann. Deep joint entity disambiguation with local neural attention. In Proceedings the 2017 Conference on Empirical Methods in Natural Language Pro- cessing, pp.
Copenhagen, Denmark, September 2017. Association for Computational Linguistics. doi: 10.18653/1//D17-1277. htRthttps:/ //wp::://wwwww.. wwwww//wwooooooo/.ooo ac aclweb. org/ prgggantology//
D1 D17-1277
Zhaochen Guo and Denilson Barbosa. Robust named entity disambiguation with random walks. Semantic Web, 9(4):499479, 2018.
Johannes Hoffart, Mohamed Amir Yosef, Ilaria Bordino, Hagen Fürstenau, Manfred Pinkal, Marc Spaniol, Bilyana Taneva, Stefan Thater, and Gerhard Weikum. Robust disambiguation of named entities in text. In Proceedings of the 201/ Conference on Empirical Methods in Natural Lan- guage Processing, pp. 782-792, Edinburgh, Scotland, UK., July 2011, Association for Computa- tional Linguistics. URL https: :ttp:s::/wwww// //www/wwwwa/////////////////// wwwwwb..ww...................... org/ httgg//oooooooo 1072
Johannes Hoffart, Stephan Seufert, Dat Ba Nguyen, Martin Theobald, and Gerhard Weikum. KORE:
Keyphrase Overlap Relatedness for Entity Disambiguation. In Proceedings of the 2 ACM International Conference on Information and Knowledge Management, CIKM '12, pp. 545-554, New York, NY, USA, 2012. Association for Computing Machinery. ISBN 97814503 1564, doi: 10.1145/239676 2396832.. Uttpts//://doi org/10.../..ooo.....ooooo 115/233/655 2396832
Chris Hokamp and Qun Liu. Lexically constrained decoding for sequence generation using grid beam search. In Proceedings of the 55th Annual Meeting of the Association for Computational
Linguistics ( (Volume 1: Long Papers), pp. 1535-1546, Vancouver, Canada, July 2017. Association for Computational Linguistics. doi: 10. 1/.18653/1/PP1-11141 URL (tthss:: www. aclweb.
org/ Maachhooooy/PP11-11
Hongzhao Huang, Larry Heck, and Heng Ji. Leveraging deep neural networks and knowledge graphs for entity disambiguation. arXiv preprint 1rXiv:1504.07674 2015.
Samuel Humeau, Kurt Shuster, Marie-Anne Lachaux, and Jason Weston. Poly-encoders: Archi- tectures and pre-training strategies for fast and accurate multi-sentence scoring. In International Conference on Learning Representations, 2020. TRL[tttps:://pperrevvee net. nww.:frrumff?
[dd S--kkknnnnVii
Jeff Johnson, Matthijs Douze, and Hervé Jégou. Billion-scale similarity search with gpus. JEEE Transactions Big Data, 2019.
Mandar Joshi, Eunsol Choi, Daniel Weld, and Luke Zettlemoyer. TriviaQA: A large scale dis- tantly supervised challenge dataset for reading comprehension. In Proceedings of the 55th An- nual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp-
11

---

Published as a conference paper at ICLR 2021
1601-1611, Vancouver, Canada, July 2017. Association for Computational Linguistics. doi:
URL htl.http::// www. htt.://...r....cooooooo///.////.
Vladimir Karpukhin, Barlas Oguz, Sewon Min, Patrick Lewis, Ledell Wu, Sergey Edunov, Danqi Chen, and Wen-tau Yih. Dense passage retrieval for open-domain question answering. In Proceedings of the 2020 Conference Empirical Methods in Natural Language Processing (EMNLP), P), pp. 6769-6781, Online, November 2020. Association for Computational Linguistics. doi: 1..186633/1/2020.cmm 2020. emn 1p main m50n.550
www.::eemmmmmmmmmmmmmmm URL btttps://www.cc.lett http:::/wwwwwc.o. wtw.://....g/aaaaoooggog.o.
Diederik Kingma and Jimmy Ba. Adam: A method for stochastic optimization. Proceedings of the 3rd International Conference on Learning Representations (ICLR), 2014.
Nikolaos Kolitsas, Octavian- Eugen Ganea, and Thomas Hofmann. End-to-end neural entity linking- In Proceedings of the 22nd Conference on Computational Natural Language Learning, pp. 519- 529, Brussels, Belgium, October 2018. Association for Computational Linguisties. doi: 10.18653/ v1/K18- 1--18811 1050. URL https: / www. ac web wwtp://o./a/aoooooooooooooo
1050
Joannis Konstas, Srinivasan Iyer, Mark Yatskar, Yejin Choi, and Luke Zettlemoyer. Neural AMR: Sequencceto--sequenee models for parsing and generation. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 146-157, Vancouver, Canada, July 2017. Association for Computational Linguistics. doi: 10.18653/v1/ P17-1014. URL wttpt:tt////m www. ac
Tom Kwiatkowski, Jennimaria Palomaki, Olivia Redfield, Michael Collins, Ankur Parikh, Chris Alberti, Danielle Epstein, Illia Polosukhin, Jacob Devlin, Kenton Lee, Kristina Toutanova, Llion Jones, Matthew Kelcey, Ming-Wa Chang, Andrew M. Dai, Jakob Uszkoreit, Quoe Le, and Slav Petrov. Natural questions: A benchmark for question answering research. Transactions of the Association for Computational Linguistics, 7:452 466, March 2019. doi: 10.1162/tacl a 00276.
URL https: wwww www.ww..wwwb.................... wwwp:eb...rggattoooooooooooo
9-126
Phong Le and Ivan Titov. Improving entity linking by modeling latent relations between mentions. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 1595-1604, Melbourne, Australia, July 2018. Association for Computational Linguistics. doi: 10. 10.18653/v1/11 8- 148. htts:tp:: /www wcwweb.
P18 1148
Phong Le and Ivan Titov. Boosting entity linking performance by leveraging unlabeled documents. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pp. 1935-1945, Florence, Italy, July 2019. Association for Computational Linguistics. doi: 10.18653/
v1/P19-1187, URI. https: www. . ac web.
Jure Leskovec, Anand Rajaraman, and Jeffrey David Ullman. Mining of Massive Dalasets. Cam- bridge University Press, USA, 2nd edition, 2014. ISBN 1107077230.
Omer Levy, Minjoon Seo, Eunsol Choi, and Luke Zettlemoyer. Zero-shot relation extraction via reading comprehension. In Proceedings of the 21st Conference on Computational Natural Lan- guage Learning (CoNLL 2017), pp. 333-342, Vancouver, Canada, August 2017. Association for Computational Linguistics. doi: 10.18653/v/KK17-1000 URL https: www aclweb. oxc/antholoogy/117
Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Veselin Stoyanov, and Luke Zettlemoyer. BART: Denoising sequence-to-sequence pre- training for natural language generation, translation, and comprehension. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pp. 7871-7880, Online, July 2020a. Association for Computational Linguistics. doi: 10. http3///11120.2mmmmmm.m 703. URL hattps://wwa http::::/wwww.llwebb... wwwpw/wwwwwwwwww..pww...p......w org/ attp//tthhott/ot/tt/ee/
acl main. 703
Patrick Lewis, Ethan Perez, Aleksandara Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich Küttler, Mike Lewis, Wen-tau Yih, Tim Rocktäschel, et al. Retrieval-auu augmenttd gener- ation for knowledge-intennivee NLP tasks. arXiv preprint arXiv:2005. /40/, 2020b.
12

---

Published as a conference paper at ICLR 2021
Jaangming Liu, Shay B. Cohen, and Mirella Lapata. Discourse representation structure parsing. In Proceedings of the 56th Annual Meeting of Association for Computational Linguistics (Volume 1: Long Papers), pp. 429-439, Melbourne, Australia, July 2018. Association for Computational Linguistics. doi: 10. 8153/v1/P188 1040. URL. httttttts:// /www. ac web. www./rrttootooooooe o
P18-1040)
I Lajanugen Logeswaran, Ming- Ming-Wei Chang, Kenton Lee, Kristina Toutanova, Jacob Devlin, and Honglak Lee. Zero-shot entity linking by reading entity descriptions. In Proceedings of the 57th Anmual Meeting the Association for Computational Linguistics, pp. 3449-3460, Florence, Italy, July 2019. Association for Computational Linguisties. doi: 10. 11.1186/1//199 1335. URL
https: www acl web. wwwporrtagggggggoooooooo PI 9- 9-1335
Pedro Henrique Martins, Zita Marinho, and André T. Martins. Joint learning of named entity recognition and entity linking. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics: Student Research Workshop, pp. 190-196, Florence, Italy, July 2019. Association for Computational Linguistics. doi: N...1653/11/11922 www. www.wb.www..ooo.. wwg. wttp://///.....oooeee1--------00
URL https:
Luca Massarelli, Fabio Petroni, Aleksandra Piktus, Myle Ott, Tim Rocktäschel, Vassilis Plachouras, Fabrizio Silvestri, and Sebastian Riedel. How decoding strategies affect the verifiability gen- erated text. arXiv preprint arXiv: 01119911005580 1.0557,, 2019.
Andrea Moro, Alessandro Raganato, and Roberto Navigli. Entity linking meets word sense dis- ambiguation: a unified approach. Transactions of the Association for Computational Linguis- tics, 2:231-244, 2014. doi: 10.1 11166//aclaa000170 URL https: /tpp:://wwww.t ac web. orrg/
anthology/Q14 1019
Isaiah Onando Mulang'. Kuldeep Singh, Chaitali Prabhu, Abhishek Nadgeri, Johannes Hoffart, and Jens Lehmann. Evaluating the impact of knowledge graph context on entity disambiguation mod- els. In Proceedings of the 29th ACM International Conference on Information & Knowledge Management, pp. 2157-2160, 2020.
Rodrigo Nogueira, Zhiying Jiang, Ronak Pradeep, and Jimmy Lin. Document ranking with pre- trained
model. Findings of the Association for Computational Lin- guistics: EMNLP 2020, pp. 708-718, Online, November 2020. Association for Computational Linguistics. doi: 10. http:////22.....gs-eeeeme mnthology/2020.oo wtt.:22.....nnioo findininneennnnnnn 63
URL https: www. acl web org/
Andrea Giovanni Nuzzolese, Anna Lisa Gentile, Valentina Presutti, Aldo Gangemi, Darío Garigliotti, and Roberto Navigli. Open knowledge extraction challenge. In Semantic Web Evalu- ation Challenges, pp. 3-15. Springer, 2015.
Yasumasa Onoe and Greg Durrett. Fine-grained entity typing for domain independent entity linking. In AAAI, pp. 8576-8583, 2020.
Myle Ott, Sergey Edunov, Alexei Baevski, Angela Fan, Sam Gross, Nathan Ng, David Grangier, and Michael Auli. fairseq: A fast, extensible toolkit for sequence modeling. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics (Demonstrations), pp. 48-53, 2019.
Fabio Petroni, Tim Rocktäschel, Sebastian Riedel, Patrick Lewis, Anton Bakhtin, Yuxiang Wu, and Alexander Miller. Language models as knowledge bases? In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Confer- ence Natural Language Processing (EMNLP-IJCNLP) pp. 2463-2477, Hong Kong, China, November 2019. Association for Computational Linguistics. doi: 10.18653/v1/D19--2255 1250. URL https: www. www.wwwwwwb......... org/ wwrgg//nthooggy.. /Dl////////.. DD/-D1911111 1250
Fabio Petroni, Patrick Lewis, Aleksandra Piktus, Tim Rocktäschel, Yuxiang Wu, Alexander H. Miller, and Sebastian Riedel. How context affects language models' factual predictions. In Au- tomated Knowledge Base Construction, 2020a. URL httts:://// /ppp//peeeeiieeeeeeeeeeeeee
/ net/forum?
1d=025x0zPfn
13

---

Published as a conference paper at ICLR 2021
Fabio Petroni, Aleksandra Piktus, Angela Fan, Patrick Lewis, Majid Yazdani. Nicola De Cao. James Thome, Yacine Jernite, Vassilis Plachouras, Tim Rocktäschel. et al. KILT: a Benchmark for Knowledge Intensive Language Tasks. arXiv preprint atXhv:2009.02252 2020b.
Francesco Piccinno and Paolo Ferragina. From tagme to wat: A new entity annotator. In Pro- ceedings of the First International Workshop on Entity Recognition and Disambiguation, ERD "14, pp. 55-62, New York. NY, USA. 2014. Association for Computing Machinery. ISBN 9781450330237. doi: 10.1145/263321 2634350.
URL https: httts:://ac..rqg/// wrg/o0....... 1145/
2633211 2634350
Matt Post and David Vilar. Fast lexically constrained decoding with dynamic beam allocation for neural machine translation. In Proceedings of the 2018 Conference of the North American Chap- ter of the Association for Computational Linguistics: Human Language Technologies, Volume (Long Papers), pp. 1314-1324, New Orleans, Louisiana, June 2018. Association for Com- putational Linguistics. doi: 10. 10.18853/11/N18-111 1119.
URL https: www. ac web. org/
anthol oithhoooyy/Nleeee 03y//18-111/1111
A. Radford, Jeffrey Wu, R. Child, David Luan, Dario Amodei, and Ilya Sutskever. Language models are unsupervised multitask learners. Technical report, OpenAI, 2019.
Jonathan Raiman and Olivier Raiman. Deeptype: multilingual entity linking by neural type system cvolution. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 32, 2018.
Michael Röder, Ricardo Usheck, Sebastian Hellmann, Daniel Gerber, and Andreas Both. - collection of datasets for named entity recognition and disambiguation in the nlp interchange format. LREC, pp. 3529-3533, 2014.
Michael Röder, Ricardo Usbeck, and Axel-Cyrille Ngonga Ngomo. Gerbil--eenhmmrking named entity recognition and linking consistently. Semantic Web, 95):6055622, 2018.
Stephen Roller, Emily Dinan, Naman Goyal, Da Ju, Mary Williamson, Yinhan Liu, Jing Xu, Myle Ott, Kurt Shuster, Eric Smith, et al. Recipes for building an open-domain chatbot. arXiv preprint arXiv:2004. 13637, 2020.
Subendhu Rongali, Luca Soldaini, Emilio Monti, and Wael Hamza. Don't parse, generate! a se- quence to sequence architecture for trie-triented semantic parsing. Proceedings of The Web Conference 2020, WWW '20, pp. 2962-2968. Association for Computing Machinery, 2020.
Hamed Shahbazi, Xiaoli Z Fern, Reza Ghacini, Rasha Obeidat, and Prasad Tadepalli. Entity Entity-aware aware ELMo: Learning Contextual Entity Representation for Entity Disambiguation. arXiv preprint 19Xi:11908.0562.. 2019.
Bill Slawski, Sep 2015.
URL httees//ww.eelycceee...... http://eheeeeeccmmm00000555/5 in -onn-uueree--eee and paddspages///
Nitish Srivastava, Geoffrey Hinton, Alex Krizhevsky, Ilya Sutskever, and Ruslan Salakhutdinov. Dropout: A simple way to prevent neural networks from overfitting. Journal of Machine Learning Research, 19956)1199-1198, 2014. URL http:://jil..oog/ppppttttt
srivastaval4a.hta
Nadine Steinmetz and Harald Sack. Semantic multimedia information retrieval based on contextual descriptions. In Philipp Cimiano, Oscar Corcho, Valentina Presutti, Laura Hollink, and Sebastian Rudolph (eds.). The Semantic Web: Semantics and Big Data pp. 382-396, Berlin, Heidelberg, 2013. Springer Berlin Heidelberg. ISBN 11183-662--828--3
Ilya Sutskever, James Martens, and Geoffrey E Hinton. Generating text with recurrent neural net- works. In Proceedings of the 28th International Conference on Machine Learning (ICML), pp- 1024-, 2011.
Ilya Sutskever, Oriol Vinyals, and Quoc V Le. Sequence to sequence learning with neural networks. In Advances in neural information processing systems, pp. 3104-3112 2014.
14

---

Published as a conference paper at ICLR 2021
Christian Szegedy, Vincent Vanhoucke, Sergey loffe, Jon Shlens, and Zbigniew Wojna. Rethink- ing the inception architecture for computer vision. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 2818-2826 2016.
James Thorne, Andreas Vlachos, Christos Christodoollpouulss, and Arpit Mittal. FEVER: large-scale dataset for fact extraction and VERification. In Proceedings of the 2018 Confer- ence of the North American Chapter of the Association for Computational Linguistics: Hu- man Language Technologies, Volume 1 (Long Papers), Pp. 809-819, New Orleans, Louisiana, June 2018. Association for Computational Linguistics. doi: 10.18653/v1/N18-1007 URL. https: / www ac web wwwp:/ooggggaaoooooooooooooooooo wgy. 1074
Johannes M. van Hulst, Faegheh Hasibi, Koen Dercksen, Krisztian Balog, and Arjen P. de Vries. Rel: An entity linker standing on the shoulders of giants. In Proceedings of the 43rd Interna- fional ACM SIGIR Conference Research and Development in Information Retrieval, SIGIR '20, pp. 2197-2200, New York, NY, USA, 2020. Association for Computing Machinery. ISBN 9781450380164. doi: 10.1145/3397271.34414166
URL latps://aci.org//II
3397271. 3401416
Ledell Wu, Fabio Petroni, Martin Josifoski, Sehastian Riedel, and Luke Zettlemoyer. Scalable zero-shot entity linking with dense entity retrieval. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pp. 1397-6407, Online, Novem- ber 2020, Association for Computational Linguistics. doi: 10. 10.18653/v1/2020 ee:::0emmmlpmmme. www.eemmannlllllee---- 519. URL https::////// http:///wwwwwww..bb../..
emnlp main. 519
Yonghui Wu, Mike Schuster, Zhifeng Chen, Quoe V Le, Mohammad Norouzi, Wolfgang Macherey, Maxim Krikun, Yuan Cao, Qin Gao, Klaus Macherey, et al. Google's neural machine trans- lation system: Bridging the gap between human and machine translation. arXiv preprint arXiv:1609.08144 2016.
Ikuya Yamada, Hiroyuki Shindo, Hideaki Takeda, and Yoshiyasu Takefuji. Joint learning of the embedding of words and entities for named entity disambiguation. In Proceedings of The 20th SIGNLL Conference Computational Natural Language Learning, pp. 250-259, Berlin, Ger- many, August 2016. Association for Computational Linguistics. doi: 10.18653/v1/K11-1100 URL https: www ac web. wwtp.///atatooog.o.........
Xiyuan Yang, Xiaotao Gu, Sheng Lin, Siliang Tang, Yueting Zhuang, Fei Wu, Zhigang Chen, Guop- ing Hu, and Xiang Ren. Learning dynamic context augmentation for global entity linking. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLL) pp- 271-281, Hong Kong, China, November 2019. Association for Computational Linguisties. doi: 10.18653/v1/DI1 1ttp://////////111 1026. URL https: www ac wwtpa//ttttgggy//////aaooo 1026
Yi Yang, Ozan Irsoy, and Kazi Shefaet Rahman. Collective entity disambiguation with structured gradient tree boosting. In Pruceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies. Volume I (Long Papers), pp. 777-786, New Orleans, Louisiana, June 8a. Association for Computational Lin- guisties. doi: 10.18653/1/N118-1077 URL https: wwww//wwwwww.. c web. org/ wwg./attooooggoooooooooooooooooo N18-1071
Zhilin Yang, Peng Qi, Saizheng Zhang, Yoshua Bengio, William Cohen, Ruslan Salakhutdinov, and Christopher D. Manning. HotpotQA: A dataset for diverse. explainable multi-hop question an- swering. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing. pp. 2369-2338,, Brussels, Belgium, Detober-November 201 8b. Association for Com- putational Linguistics. doi: 10.18653/v//DIs 1259. URL https: wwwww aclweb.org/
anthology/DD8- 1259
15

---

Published as a conference paper at ICLR 2021
A EXPERIMENTAL DETAILS
We implemented, trained, and evaluate model using the fariseq library (Ot al. 2019 We trained GENRE for every task using Adam (Kingma & Ba 2014) with a learning rate 10 with a linear warm-up for 500 steps and then liner decay. The objective is sequence- -equeenec categorical cross-entropy loss with 0.1 of label smoothing.
A.1 NAMED ENTITY DISAMBIGUATION
Setting Given document dj (e.g.. a sentence) containing a set of entity mentions mi....; m.N, a system either has to assign, to each mention m, either KB entity (i.e., E E), or predicts that there is no corresponding entry in the KB (i.e., G; NIL). Moreover, restricted candidates set C. = fê.....,, S & U } for each mention m, is provided.
Training We pre-trained GENRE BLINK data for 200k steps and then do model selection on the validation set. Afterward, we fine-tuned on AIDA without resetting the learning rate nor the optimizer statistics for 10k steps and we do model selection the validation set. Following previous works (Yamada et al. [016 Ganea & Hofmannn [0177] Le & Titov, 2018). we considered only mentions that have entities in the KB (i.e., Wikipedia). Training was done on GPUs (with 32GB of memory) and it completed in ~44h for a total of ~32 GPU/day.
Inference At test time, use Constrained Beam Search with 10 beams, and maximum decoding steps of 15. We restrict the input sequence to be at most 384 tokens cutting the left, right, or both parts of the context around mention. We normalize the prooobbbbiliitis by sequence length.
A.2 ENTITY LINKING
Setting Given a document dj (e.g., sentence) system has to return set of tuples (m; ea) where m, is a entity mentions (a span contained in d,) and e; E & its corresponding entity in the KB. Following Kolitsas et (2018), we considered only mentions that have entities in the KB (i.e.. Wikipedia) and we used their candidate sets with the additions of the table computed by Hoffart al. 10011)
Training We pre-trained GENRE on all abstract sections from Wikipedis@enniiee enriched by string matching heuristic to solve co-references (i.e., if there is a string that matches exactly with another hyperlink we also add it to the dataset as a mention/entity pairs) data for 200k steps. Then we do model selection on the validation set. Afterward, we fine-tuned on AIDA resetting the learning rate and the optimizer statistics for 10k steps and we do model selection on the validation set. Again, following previous works (Kolitsas et al. 2018). 8), we considered only mentions that have entities in Wikipedia. Training was done on 64 GPUs (with 32GB of memory) and it completed in ~30h for total of 80 GPU/day.
Inference At test time, we use Constrained Beam Search with 6 beams, and maximum decoding step of 384. When the input sequence is too long, split the input into multiple chunks of equal size. We normalize the log-probabilities by sequence length.
A.3 PAGE-LEVEL. DOCUMENT RETRIEVAL
Setting Given query & (e.g., question) and collection of documents D (in KILT are Wikipedia pages), a system has to rank documents D based on their relevance to q.
Training We trained GENRE on all KILT data simultaneously for 200k steps and we do model selection on the validation set averaging the score across tasks. Training was done on 128 GPUs (with 32GB of memory) and it completed in ~33h for total of 176 GPU/day.
It is based the 2019/08/01 Wikipedia dump pre-processed by Petroni 2020b).
16

---

Published as a conference paper at ICLR 2021
Inference test time, Constrained Beam Search with 10 beams. For the ED sub-task, restrict the imput sequence to be at most 384 tokens cutting the left, right, or both parts of the context around a mention. We normalize the log- probaabbaiilsiss by sequence length.
B
ADDITIONAL RESULTS
B.1
NAMED ENTITY DISAMBIGUATTINN
Table reports evaluation of GENRE on WikilinksNED Unseen-Mentions data (Onoe & Durrett We also report additional results on AIDA from the literature in Table
2020
Seen
Unseen
Total
Exact match
87.48 (751)
70.36 (2227)
74.68 (2578)
Partial match
56.39 (1566)
61.47 (4838)
60.23 (6404)
No match
41.46 (205)
45.04 (413)
43.85 (618)
Total
64.43 (2522)
63.21 (7478)
63.52 (10R)
Table 6: Evaluation of GENRE on WikilinksNED Unseen-Mentions data (Onoe & Durrett 2020) We train on the provided train set and we report accuracy scores (i.e., precision at I) alongside with the number of supporting datapoints. We report scores splitting the test set in seen and unseen entities as well as in three different matchings between mention and its gold entity.
Methods
micro-F,
Guo & Barbosa (0018)
89
Le & Titov (2019)
89.6
Yamada et al. (2016)
91.5
Ganea Hofmann
(2077)
92.2
Shahbazı et al. 2019
93.5
Chen et (020))
93.5
Yang ct al.
2019
93.7
Fang al.
2019)
94.3
Raiman & Raiman
(0188
94.9
Mulang et al.
(020)
94.9
Yang et al. (2018a)
95.9
GENRE
93.3
Table 7: Additional results on AIDA, We report Micro InKB Fi on test sets.
B.2 DOCUMENT RETRIEVAL
Table&extennt extends Table with additional results (i.e., training GENRE the numerical identifiers) and an ablation study on the document retrieval task. The purpose of the experiment is to see whether GENRE benefits from the entity names to be meaningful as well compositional. Numerical IDs do not have that property. In both cases, the model uses its memorizing capabilities but when using IDs the performance significantly low. Indeed. with IDs the model has no way to generalize nor to use "implicit knowledge" acquired during the unsupervised pre-training We also ablate the training data. DPR data corresponds to training only on Natural Questions (NQ) and TriviaQA (TQA) as DPR was trained only for QA tasks on those datasets and two extra ones. Note that training on BLINK data corresponds to only training for entity disambiguation. However, every other task share similarities with entity disambiguation and thus the model is also capable to address the other tasks with non-zero performance. For the ablations, underlined cells indicate what are the results on the respective task on which model was trained for (i.e., GENRE only BLINK data trained only for ED where GENRE only DPR data was trained only for QA). The ablation on data suggests that it is beneficial to train all tasks simultaneously. GENRE without constraints indicates ablating
17

---

Published as a conference paper at ICLR 2021
constrained decoding which implies unconstrained generation (i.e., the model may generate entity names that are not in the KB).
Fact Chock.
Entity Disambiguation
Slot Filling T-REx
Open Domain
Dial. WoW
Model
FEV
AY2
WaWi
WnCw
zsRE
NQ
HoPo
TQA
ELI5
Avg.
DPR BERT
72.9
40.1
60.7
25.0
43.4
DPR
55.3
1.8
0.3
0.5
13.3
28.9
54.3
25.0
44.5
10.7
25.5
23.6
Uidd
50.9
3.7
0.24
2.1
44.7
60.8
28.1
34.1
46.4
13.7
49.0
30.5
DPR BART
55.3
75.5
45,2
46.9
13.3
28.9
54.3
25.0
44.4
10.7
25.4
61.9
72.6
28.7
53.7
BLINK flair
63.7
81.5
80.2
68.8
59.6
78.8
24.5
46.1
65.6
9.3
38.2
56.0
GENRE only BLINK IDs
1.8
65.0
63,5
58.6
0.1
0.2
0.4
0.3
5.4
0.3
13.3
19.0
GENRE only DFR data
70.8
9.7
1.9
7.3
60.0
79.7
58.3
40.3
69.6
13.2
52.6
42.1
GENRE only BLINK data
28.1
82.5
88.1
69.9
44.8
66.1
15.0
16.4
25.6
6.8
38.7
43.8
GENRE constraints
78.9
87.2
83.2
36.5
74.4
93.6
53.3
63.7
62.7
GENRE full
83.6
89.9
87.4
71.2
79.4
95.8
60.3
51.3
69.2
15.8
62.9
69.7
Table 8: Ablation study on KILT retrieval. We report R-Precision. GENRE only BLINK IDs denotes training BLINK (Wu et al. 2020) data where instead of using the textual entity represen-
tation as target we used a numerical ID.
17ex04
Accuracy
Data disntbucion
57
1 12e404
accur
1.e+04
0.
5.0e103
33c103
10
11
12
13
14
Number of Tokens
Figure 4: Accuracy per number of BPE tokens of the Wikipedia title generate on the validation sets of all KILT datasets except ELI5 (as it is fundamentally different from the others). We also show the data distribution token lengths. Most the titles have less than BPE tokens while the mode of the distribution is 5. Here GENRE has average accuracy of 78.6% but it is higher for short titles (c.g., <10) and it is lower for long titles (e.g., 10). Degradation in performance does not directly follow the data distribution of the token lengths. Indeed, even if long titles rare performance not heavily affected (e.g., for length > 15)..
18

---

Published as a conference paper at ICLR 2021
Acturacy
disrloution
15e+0e
3.
133404
97
12c104
accurd
10c104
6.c102
50e102
33e403
No
21
2'
20 2
0.ee00
Number of incoming links Wikipedia
Figure 5: Accuracy per number of incoming links in Wikipedia on the validation sets of all KILT datasets except ELI5 (as it is fundamentally different from the others). We also show the data distribution the number incoming links. Intuitively, a page/entity with few incoming links has been observed less than highly connected pages/entities. Indeed, for pages/entities bin on the left) the average accuracy is 20% lower than the global average (78.6%). However, for pages/entities linked at least once it is above the global average. This indicates that GENRE seems effective on linking rare entities.
linked (first
C EXAMPLES
ID: 87d95287-707e-444 ca03-1 1a4a3a. World Without. Superma::
inputs: "[.-] When Superman leaves Earth for New Krypton he appoints , newly freed from the Phantom Zone to take his place as guardian of [START.ENT] Metropolis (END,ENT the
Johnathan Kent tribute Clark \ adoptive father posing as Clark \'s cousin [..] gold_output 'Metropolis (comics)"
Mon-EI
identity
predicted.outputs:
('Metropolis (comics)', -0.99).
('Themyscira. (DC. Comics) -1.09),
("Metropolis. pittropoisssdisaaene (disambiguation)
-1.27).
('Superman_ (comic.booo -1.51).
("Superman. ( (Earth-Two)- -1522)
]
Figure 6: Example of a GENRE prediction for named entity disambiguation on KILT WNED. The input is plain text where mention is flagged with special start and end tokens (START ENT] and (END_ENT). The output is ranked list of entity (where we report the log-likelihood as well).
ID: sfq_18225
inputs: 'Which Florentine painter
1535-1607 used the name Bronzino after the death of his 'uncle "
ID: '4713'
inputs 'Tool won three Oscars.
gold output: Bronzino
gold output: 'Toal (band)
predicted outputs:
predicted outputs: TTool (band)' 0.000,
('Fiorence -0.77)
('Bronzino -0.62)
(Tooll disambiigationn
-1559).
('Niccolo Machiavelli -0.44).
Machine Head. (band) -1.73)
('Giorgio_de, dww_Chirrccoooooo -0.11).
LLanguage_Arte (album) -1.97).
('Vitruvian.Man -0773)
"Machine.Gun.
-2.12)
10
]
10
]
(a) TriviaQA (open domain question answering).
(b) FEVER (fact checking).
Figure 7: Example of GENRE predictions for the retrieval task on KILT. The input query and the output a ranked list of Wikipedia article titles (we also report the log-likelihood of the solutions).
19

---

Published as a conference paper at ICLR 2021
ID: (110testa_SOCCCER
inputs: 'SOCCER RESULT SPANISH FIRST DIVISION. MADRID 1996-08-31 Result of game
played in the Spanish tirst division on Saturday: Deportivo Coruna 1 Real Madrid i.
gold output: 'SCCCER
[ Spain] FIRST DIVISION [MADRID] Madrid) 1996-08-31 Result of game played in the [Spanish]
first division Saturday
Deportivo Coruna 1 [Real [adrid][[R)) Real Madrid C.F.)
predicted .ouputt 'SOCCER RESULT IN [SPANISH][ Spain) FIRST DIVISION [MADRID] Madrid)
1998 08 31 Result of game played the SSpanish (Spain) first division on Saturday
[Deportivo [( Deportivo
Coruna) Coruna [Real Madrid] Real Madrid C.F.) 1.
gold spans:
[19, 'Spain"
[44. 6. 'Madrid")
[91. "Spain
[147, 11, Meal
10
1
palicted_spanne
12
[19, 7. "Spain'
13
[44, 'Madrid )
14
[91. 'Spain
15
[128. 9. Deportivo.de Coruna '],
16
[147, 11, MRaal_MddridC.FF.
17
]
18
19 : 0.80
20
Micro-recall
1.00
21
Micro-F1:
0.88
Figure 8: Example of GENRE prediction for end-to-end entity linking on AIDA. The input is plain text and the output is Markup string where the links are Wikipedia titles. Spans are in the format (st, le ti): start of the mention, length of the mention, and title respectively.
English
France
language
literature
EOS
BOS
EOS
Figure 9: Example of prefix tree (trie) structure where the allowed entities identifiers 'English language', 'English literature' and 'France'. Note that at the root there is the sfart-of-sequence token SOS and all leaves end-of-sequence tokens EOS. Since more that one sequence has the same prefix (i.e., "English'), this end up being an internal node where branches are the possible continuations.
20


Reviews:
[{"review_id": "gmngTl8-zQe", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 7, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper proposes a brand new approach for entity retrieval, which leverages an encoder-decoder architecture to generate the target entity directly. To tackle the problem of invalid generation, a trie-constrained beam search is used for decoding. The author performs intensive experiments on entity disambiguation, entity linking, and document retrieval tasks and achieves new SOTA or competitive results on over 20 datasets. Although the paper does not come up with new architecture or elaborately designed neural components, I believe this paper is worth reading for the community, including how this paper redefines the problem.\n\nStrengths:\n1. A brand new perspective on entity retrieval. Clever use of text generation methods.\n2. Compared to previous methods, GENRE is memory saving.\n3. This paper provides comprehensive experiments on 3 important tasks, over 20 datasets, showing the robustness of this method. \n\nWeaknesses: \n1. Lack of clarity at many places: \na) It's clearer to demonstrate how you reformulate the input and output for the 3 tasks, as they seem very different compared to previous methods\nb) It's better to add some description about the baseline systems\nc) I'm especially interested in the decoding details on end-to-end entity linking,  it's better to use a formular to demonstrate how you compute the log-probabilities.\n\nQuestions:\n1. In the entity disambiguation task, it seems that for each inference step, GENRE only generates one entity. Is it possible to generate all entities simultaneously as what GENRE does in end-to-end entity linking? \n2. From table 5, an impressive result is that it seems that the pre-trained model is good at memorizing, even if the identifier is numeric. But from table 3, some of the baseline systems also benefit from pretrained models, while achieve much lower results. What do you think causes this difference? \n", "extra_scores": {}, "extra_text": {}}, {"review_id": "YPITSP2P0IC", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper introduces a new method to retrieve entity by auto regressively generating unique entity name as a sequence of word pieces, instead of pinpointing the ID representing an entity. This method stands out in novelty compared to existing various entity retrieval methods, which always assigns a single ID to each entity. Practically, the proposed method has two nice properties: (1) When the entity vocabulary is very large, this approach requires less parameter space and memory compared to other methods (as shown clearly in Table 4) (2) The model can address novel entities, which was unseen during the training. The paper is clearly written and extensively evaluated on three relevant tasks, entity disambiguation, entity linking, and entity retrieval.\n\nI have one big concern with the current format of the presentation. In the current draft, It is not very clear whether the strong gain is coming from large scale pretr aining or the proposed method itself.  To this end, the ablations shown in Table 7 and Table 8 should be reported in the main paper, with clear explanations. As we all know, the model architecture (which is the focus of this paper) cannot be properly evaluated when the training set up is different (i.e., how much pretraining has been done, on what dataset?).  Could you elaborate on this? In appendix, there's only result tables without in-line explanations. \n\nThe experiments on cold start, as well as table 5 which shows performances on entities divided by name match is pretty cool!\n\nIf space allows, adding some more analysis on what types of entities do this model do better compared to other methods would be interesting (would lengthier names easier or harder? would it do better on popular entities or more long tail ones? are these systems complementary to existing methods or mostly succeed and fail on the same set of examples?) \n\n ", "extra_scores": {}, "extra_text": {}}, {"review_id": "MdMUmOHa5d", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "### Summary\n\nThis paper proposes to tackle the entity linking task using a sequence-to-sequence neural model, trained by producing unique entity names, in autoregressive fashion. The paper makes a case that this approach can scale better with larger entity vocabularies than previous methods with dedicated entity representations both in terms of memory as well as computation costs. The model is studied under a number of tasks including entity disambiguation, entity linking and document retrieval for question answering.\n\n### Strong and weak points\n\nStrong points:\n- The work is well motivated: I believe there are, or will be, many systems where retrieving information about billions of entities is useful. Making simpler and more efficient models to deal with larger entity vocabularies is important.\n- Doing seq2seq entity linking is really a novel idea and is surprising that it works so well for several of the datasets presented.\n- The idea of constrained decoding using a Trie is neat and makes intuitive sense.\n- The empirical evaluation in the paper is quite exhaustive, in terms of number of datasets.\n\nWeak points:\n- There is no discussion or analysis on the performance of this new model on “tail entities” (entities that have few examples in the training set). A believe such a discussion would be interesting for two reasons: (1) if one wishes to use this type of model for much larger entity vocabularies, it is likely that a larger fraction of entities will have low number of examples, and (2) one effect of contrastive learning (e.g., negative sampling) has on systems with explicit entity representations is some implicit training of _all_ entities, which is lacking in the described autoregressive proposal.\nFinally on this topic of tail entities: the “IDs” experiment, shown in Table 5, indicates that when entity mention and decode target are not related, performance suffers. Similarly, for “tail entities” that are ambiguous with another “popular entity”, the model may be biased with the popular entity (results from “Cold Start” may support this hypothesis). Hard to say without some analysis.\n\n- While the constrained decoding is really interesting, it was not clear from the paper how crucial this was for good performance. Is this something absolutely critical for performance overall, or does it provide a modest performance improvement? While it may be clear to the authors, it would be highly informative to explicitly describe the performance impact on unconstrained beam decoding.\n\n### Recommendation\n\nOverall, my recommendation is to accept this paper to ICLR. I believe the problem of representing entities in natural language systems is of high interest to the ICLR and NLP communities. The ideas in this paper are novel, the paper is well written and the empirical evaluation is well executed.\n\n### Questions for authors\n\n- See comment above regarding analysis of “tail entities”. Could any further insights on this topic be added to the paper? \n- See comment above about ablating the constrained beam decoding. This would not need to be a long or complex analysis. Simply 1 or 2 data points for the reader to understand the magnitude of the importance of this decoding.\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "JMSBRkLIT5i", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper proposed to use autoregressive approach to solve entity-based problems. They proposed a uniform framework and showed that their model achieved the state of the art performance on 3 different types of tasks (~20 datasets). The GENRE model also significantly reduced the memory usage compared to previous models that stored a big memory table. It's also capable of linking novel entities at inference time. This paper is clearly written. The experiment results are convincing.\n\nOne limitation of this paper is that they required the vocabulary of the entities to be the ones that have a Wikipedia page. And that their model relied on copying the surface form of entities (as suggested in the paper). From the experiments, the \"copying\" approach worked very well on Wikipedia entities, that are often common entities. How can you improve the model to link rare entities not in the Wikipedia pages? \n\nAnother concern is the efficiency at inference time. Compared to the models with a large entity memory whose retrieval is performed with Maximum inner product search, how is the efficiency of your decoding strategy?", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This is a novel, simple, and experimentally well-supported new idea for entity linking.  The key insight is to perform entity linking by producing meaningful entity names with seq2seq approaches, and the big surprise is how well this works experimentally (at least for wikipedia-style entities).  Very nice paper!

Decision: Accept (Spotlight)