EntQA: Entity Linking as Question Answering
Authors: Wenzheng Zhang
Wenyue Hua
Karl Stratos
Venue: ICLR 2022 Spotlight (2022)
Keywords: Entity linking
open-domain question answering
dense retrieval
reading comprehension
information extraction
natural language processing

Abstract: A conventional approach to entity linking is to first find mentions in a given document and then infer their underlying entities in the knowledge base. A well-known limitation of this approach is that it requires finding mentions without knowing their entities, which is unnatural and difficult. We present a new model that does not suffer from this limitation called $\textbf{EntQA}$, which stands for $\mbox{\textbf{Ent}ity}$ linking as $\mbox{\textbf{Q}uestion}$ $\mbox{\textbf{A}nswering}$. EntQA first proposes candidate entities with a fast retrieval module, and then scrutinizes the document to find mentions of each candidate with a powerful reader module. Our approach combines progress in entity linking with that in open-domain question answering and capitalizes on pretrained models for dense entity retrieval and reading comprehension. Unlike in previous works, we do not rely on a mention-candidates dictionary or large-scale weak supervision. EntQA achieves strong results on the GERBIL benchmarking platform.


Published as a conference paper at ICLR 2022
ENTQA: ENTITY LINKING AS QUESTION ANSWERING
Wenzheng Zhang, Wenyue Hua, Karl Stratos Department of Computer Science
Rutgers University (wenzheng. zhang, wenyue. hua, karl www..sssratosserrttege. edu
ABSTRACT
A conventional approach to entity linking is to first find mentions in given docu- ment and then infer their underlying entities in the knowledge base. A well-known limitation of this approach is that it requires finding mentions without knowing their entities, which is unnatural and difficult. We present new model that does not suffer from this limitation called EntQA, which stands for Entity linking as Question Answering. EntQA first proposes candidate entities with fast retrieval module, and then scrutinizes the document to find mentions of each candidate with powerful reader module. Our approach combines progress in entity linking with that in open-domain question answering and capitalizes on pretrained models for dense entity retrieval and reading comprehension. Unlike in previous works, we do not rely on mention mention-candidates candidates dictionary or large-scale weak supervision. EntQA achieves strong results on the GERBIL benchmarking platform.
INTRODUCTION
We consider the most general form of entity linking (EL) in which a system, given document, must both extract entity mentions and link the mentions to their corresponding entries in knowledge base (KB). EL is foundational building block in automatic text understanding with applications to question answering (QA) (Ferrucci) (012)). information retrieval (Xiong et al.) [017]] Hasibi et al.)
2016
Balog et al. 2013, Reinanda al.) 2015), and commercial recommendation systems (Yang
et al. 2018 Slawski 2015).
The output space EI. intractably large. Any subset of all possible spans in the document linked to any KB entries (typically in the order of millions) can be a system output. To get around the intractability, existing methods decompose EL into mention detection (MD) and entity disambigua- tion (ED) and tackle them with varying degrees of independence. In all cases, however, the order of these two subproblems is MD followed by ED: first the system identifies potential entity mentions, and then the mentions are resolved to KB entries. Previous works either assume that mentions given (Gupta et al. 2017). run an off-the-shelf named-entity recognition (NER) system to extract mentions and resolve them by ED (MD+ED pipeline) (Hoffart al. 2011 Ling al., 2015 van Hulst et al., 2020). or train an end-to-end model that jointly performs MD by beam search (Kolitsas et al. 2018 De Cao al. 2021).
A limitation of performing MD before ED is that it requires finding mentions without knowing the corresponding entities. By definition, a mention needs an entity (i.e., mention of what?). Existing methods suffer from the dilemma of having to predict mentions before what they refer to, which unnatural and difficult. For example, the MD ED pipeline heuristically extracts mentions from spans of named entities found by a third-party NER system, and the performance bottleneck is often errors in MD propagated to ED. End-to-end models alleviate the problem of error propagation, but the search is only approximate and the dilemma, albeit to lesser degree, remains.
In this work, we propose flipping the order of the two subproblems and solving ED before MD. We first find candidate entities that might be mentioned in the given document, then for each candidate find its mentions if possible. Our key observation is that while finding mentions is difficult without the knowledge of relevant entities, finding relevant entities is easy without the knowledge of their specific mentions. This simple change fundamentally solves the dilemma above since identifying mentions of particular entity is well defined.

---

Published as a conference paper at ICLR 2022
We cast the problem as inverted open-domain QA. Specifically, given document, we dual encoder retriever to efficiently retrieve top-K candidate entities from the KB as "questions". Then apply a deep gross-attention reader on the document for each candidate to identify mentions of the candidate in the document as "answer spans". Unlike in standard QA, the model must predict an unknown number of questions and answers. We present simple and effective solution based on thresholding. We call our model EntQA, standing for Entity linking as Question Answering.
Beyond conceptual novelty, EntQA also offers many practical advantages. First, EntQA allows us to piggyback on recent progress in dense entity retrieval and open-domain QA. For instance, we warm start EntQA with the BLINK entity retriever (Wu et al. 2020a) and ELECTRA finetuned on a QA dataset (Clark et al. (019) to obtain an easy improvement. Second, EntQA has no dependence on a hardcoded mention candida-- dictionary which used in previous works to reduce the search space and bias the model (Ganea & Hofmann 2017 Kolitsas et al., 2018 De Cao et al. 2021 The dictionary is typically constructed using a large KB-specific labeled corpus (e.g., Wikipedia hyperlinks), thus having no dependence on it makes our approach more broadly applicable to KBs without such resources. Third, training EntQA is data efficient and can be done with an academic budget, in contrast with GENRE (De Cao et al. 2021) which requires industry-scanl pretraining by weak supervision.
EntQA achieves strong performance the GERBIL benchmarking platform (Röder et al. 2018)
The in-domain F1 score on the test portion of the AIDA-CoNLL dataset is 85.8 (2,1 absolute im- provement). The macro-averaged F1 score across 8 evaluation datasets is 60.5 (2.3 absolute im- 1 provement) We analyze EntQA and find that its retrieval performance is extremely strong (over 98 top-100 recall on the validation of AIDA), verifying our hypothesis that finding relevant entities without knowing their mentions is easy. We also find that the reader makes reasonable errors such as accurately predicting missing hyperlinks or linking a mention to a correct entity that more specific than the gold label.
2
MODEL
Let denote the set of entities in KB associated with text title and description. Let V denote the vocabulary and X {i E VT, :1 < T < Tmax} the set of all documents up to length Tmax- EL is the task of mapping x X' to E (()(a)) where V(x) {(s, t, t)e S < t S | E E} the set of all possible linked spans in x and P is the power set. The size of the output space is (2 where [8] is typically very large (e.g., around 6 million in Wikipedia) and Tmax can also be large (e.g.. > 3000 in AIDA), ruling out any naive exhaustive search feasible approach.
EMQA decomposes EL into two subproblems entity retrieval and question answering. More specif- ically, given document X,
1. The retriever module retrieves top-K candidate entities that might be mentioned in æ.
2. The reader module extracts mentions of each candidate entity in x (or rejects it), then returns a subset of globally reranked labeled mentions as the final prediction.
Our approach bears superficial similarities to standard framework in open-domain QA that pipelines retrieval and span finding (Karpukhin et al., 2020 inter alia), but it has the following important differences. First, instead of retrieving passages given a question, it retrieves questions (i.e., candidate entities) given a passage. Second, even when considering a single question, there can be multiple answer spans (i.e., mentions) instead of one. Both the number of gold entities present in a document and the number of mentions of each gold entity are unknown, making this setting more challenging than standard QA in which we only need find single answer span for a single question on a passage.
Input representation. Both the retriever and the reader work with text representations of docu- ments and entities, thus applicable to zero-shot setting (c.g., linking to new KB at test time by reading entity descriptions). We use the title (e) E V+ and the description (dese(e) E V to represent an entity E E. Since document x E X' is generally too long to encode with Trans- former encoder which has quadratic dependency on the input length, we break it down in my E N
"ode available at: httpps///giittaa..../////eeeeeee
2

---

Published as a conference paper at ICLR 2022
overlapping passages Pı (z) ... Pm_ (m) E V of length L with stride S (e.g.. L = 32 and S 16) and operate at the passage-level similarly as in QA (Alberti et al. 2019). When a document is long, individual passages may lose global information. For long documents, we find it beneficial to carry a document-level topical text Propic (x) E V across passages in that document (e.g.. first sentence). We emphasize that do not use any extra information outside the document. In experiments we simply set Propii (x) = I1 E V (i.e., the first token in the document).
Notation. We write enc ya ÆaxT denote a Transformer encoder that maps any token sequence to the same-length sequence of corresponding contextual embeddings; the symbol S is used distinguish different encoders. We assume the usual special tokens in the input popularized by BERT (Devlin et al. 2019): [CLS] to represent the whole input and [SEP] to indicate an input boundary. We write denote the concatenation; we insert an unused token type in the vocabulary in between two texts being concatenated. We write M, E IRd to denote the i-th column of matrix M RRdXT"
2.1
RETRIEVER
Given a passage E 4 in document and an entity E E, the retriever computes
P enc (ICLS) p SEP (2))
Ea ency 2 [CLS] (e @ @deso ( (SEP])
score zete (p, I, E) P1 ET
At inference time, we precompute E Rd for each e E & and use Faiss (Johnson et al. 2019) for fast top-K retrieval.
Training. We train the retriever by multi-label variant of noise contrastive estimation (NCE). Given passage p in document x, we have a set of multiple gold entities {(p) C & that are mentioned in the passage and optimize the per-example objective
exp (scoref fetr (D,x,e)
Σ log cEE(p)
max 8
exp (score (p, e)) + Ee'EN( exp (score (D, x, e²))
where N(E, p) c (p)) set of negative examples that excludes all gold entities (p). The objective effectively constructs {E()) independent NCE instances, each of which treats a gold en- tity as the only correct answer while ensuring that other gold entities are not included in negative examples. We obtain 90% of N(E,p) by sampling entities uniformly at random from E and 10% by hard negative mining (i.e., using highest-scoring incorrect entities under the model), which well known be beneficial entity retrieval (Gillick et al., 2019 Wu al. 2020a Zhang & Stratos 2021)
2.2 READER
Let 61:k (e1 ... eK) gK denote K candidate entities for passage p in document For each E (1... K). the reader computes a joint encoding of (p, x. 6x) by
# = enc [CLS]pe drropicz)) [SEP] (eu) O @desc (e4) [SEP])
then defines conditional distribution over mention spans of ez in p by
exp (wTtrtt H
(spp
As E (1... 1pl
Σ 1pl +1 exp (wTan Hb)
Di=1
exp (w (wTH H²)
(tap, = 1pl+1 exp (w H+)
Vt E (I...|pp|+ +p]
Elr
Popan (8, 11p,2r, (xp,are))))))) = Potan (s/p, R. ex) x p xt)
Vs,t (1...p|+1 1|
3

---

Published as a conference paper at ICLR 2022
where Watart; Wend R additional parameters. The reader also multitasks reranking: it uses Wrerank E Rd to define a conditional distribution over candidate entities by
exp (W Terank H
" (eepankes/pp,zee..
Ak [1...KK
Σ ===11 exp (w. rerank H
Training. We obtain candidates EiKK from a fully trained retrieval module to make training con- sistent with test time, During training, we always include all gold entities as candidates (i.e., (()) C E1.K). Let M(p,e) e) denote the set of gold mention spans of E & in pł if is not present in P. we define M(p.e) e) = ((1,)); We optimize the per-example objective
K
max 8
Σ
lex E E()) log Pferank (ex p, x, G1:k) +
Σ (s.c).M(p:ca)
log log (8, t/p, ex) (2)
where ((4) is the indicator function equal to one A is true and zero otherwise. Note that the reader is trained to predict the [CLS] span for incorrect entities.
2.3 INFERENCE
At test time, we process new document x E X' in passages E VL independently follows:
1. Retrieve top-K K highest scoring entities 61:K under score (p, x, e).
2. For each candidate k, extract top-P most likely mention spans (s (i..t)... (sp, ($) under Pspan (s, tlpp Ir Ex) while discarding any span less probable than 11,
3. Return subset of the surviving labeled mentions (s,t, ex) with Plerank (ex I, c1.KK x Popaa (s, t/p, xr, ex) > 7 as the final prediction.
We do not apply any further processing to combine passage-level predictions other than merging duplicate labeled spans (s, t, e) in the overlapping sections. This inference scheme simple yet effective. For each candidate entity, the reader scrutinizes the passage with deep cross-attention to see there are any mentions of the entity and has a chance to reject it by predicting (1, 1). The reader delays its final decision until it has processed all candidates to globally reconsider labeled mentions with ranking probabilities. Figure [ shows a successful prediction on a passage from the validation portion of AIDA.
EXPERIMENTS
We evaluate EntQA the GERBIL benchmarking platform (Röder et al.) 2018). which offers reli- able comparison with state-of-the-art EL methods on numerous public datasets.
3.1 SETTING
Datasets. We follow the established practice and report the InKB Micro F1 score on the in-domain and out-of-domain datasets used in De Cao et al. (20IT). Specifically,
the AIDA-CoNLL dataset (Hoffart et al. 201T) as the in-domain dataset: we train EntQA on the training portion of AIDA, the validation portion (AIDA-A) for development, and reserve the test portion (AIDA- B) for in-domain test performance. We use seven out-of-domain test sets: MSNBC, Derczynski (Der) (Derczynski al., 2015), KORE 50 (K50) (Hoffart al. 2012), N3-Reuters-125 (Rl 18)28), N3-RSS-500 (R500) (Röder et al. 2014), and OKE challenge 2015 and 2016 (OKE15 and OKE16) Nuzzolese et 2015). We refer to Table 6 in Kolitsas al. (0018) for the datasets' statistics. For the KB, we use the 2019 Wikipedia dump provided in the KILT benchmark (Petroni et al.) 202
which contains 5.9 million entities.
Model details. We initialize the passage encoder eec and the entity encoder ench e in the retriever module with independent BLINK retrievers pretrained on Wikipedia hyperlinks (Wu et al.) 2020a) and optimize the NCE objective (1) with hard negative mining. We initialize the joint encoder
4

---

Published as a conference paper at ICLR 2022
Top-K candidate entities
Leieestershire County Cricket Club
2. Grace Road
Passage
S.mersstt County Cricket Club
X
4. Durham County Cricket Club
X Nottinghamshire County Cricket Club
After bowling [Somerset]s out for 83 on the opening morning at [Grace Road]2, [Leicestershire]] extended their first in- nings by 94 runs before being bowled out for 296 with [England]ı
X Derbyshire County Cricket Club
X Warwickshire County Cticket Club
K Leicestershire
X 9. Worcestershire County Cricket Club
X 10. Yorkshire County Cricket Club
11. England cricket team
12. Marylebone Cricket
K 13. Sussex County Cricket Club
K 14. Kent County Cricket Club
15. Leicester
16. Aylestone Road
K 17. County Cricket Ground, Derby
Figure 1: Example prediction by EntQA taken from AIDA-A. Given passage, the retriever mod- ule ranks K candidate entities, then the reader module finds mentions each entity or rejects (marked by x). Both modules use entity descriptions (not shown). In this example, it predicts the span "England" for the I 1th candidate Engl and cricket team but rejects the 35th candidate England (the country).
ency in the reader module with ELECTRA- Alage (Clar et al. 2019) finetuned on SQuAD 2.0 (Rajpurkar al. 2018) and optimize the reader objective (2). We break up each document X into overlapping passages of length 7. 32 with stride 5 = 16 under WordPiece tokenization. For each passage in æ, we concatenate the input with the first token of the document (x) = 21, which corresponds to the topic in AIDA but not in other datasets. We use 64 candidate entities in training for both the retriever and the reader; we use 100 candidates at test time. We predict up to P 3 mention spans for each candidate entity. We use 0.05 the threshold in all experiments, chosen after trying values 0,01, 0.1, and 0.05 on the validation set. Additional experiments on automatically tuning y are discussed in Appendix A For optimization,
Adam (Kingma & Ba (013) with learning rate 2e-6 for the retriever and le-5 for the reader; we use a linear learning rate decay schedule with warmup proportion 0.06 over epochs for both modules. The batch size is 4 for the retriever and 2 for the reader. The retriever is trained on 4 GPUs (A100) for 9 hours; the reader trained on 2 GPUs for hours.
Baselines. We compare with state-of-the-ar EL systems that represent diverse array of ap- proaches. Hoffart et al. (2011) and van Hulst et al. (2200) use the MD)>ED pipeline; despite the limitation of pipelining MD with ED, the latter achieve excellent performance by solving MD with a strong NER system (Akbik et al. 2018). Kolitsas et al. (2018) use an end-to-end model that sequen- tially performs MD and ED; to make the problem tractable, they drastically prune the search space with mention-candidates dictionary and the model score. [De Cao et al. (202T) propose GENRE, a to-nnece-----euunceee model for EL. The model conditions on the given document and autore- gressively generates labeled version of the document by at each position either copying token, starting or ending a mention span, or, if the previous generation was the end of a mention m, gener- ating the entity title associated with m token by token. At inference time, GENRE critically relies on a prefix tree (aka. trie) derived from Wikipedia to constrain the beam search so that it produces a valid entity title in the KB. Since each beam element must first predict mention before predicting an entity, unless the beam size is unbounded so that every labeled span is considered, GENRE will suffer from MD emrors propagating ED.
3.2 RESULTS
Table I shows the main results. EntQA achieves the best in-domain test F, score for AIDA (+2.1) and is also performant out-of-domain datasets (+3.8 KORE 50 and +7.4 N3-Reuters- 128, close second-bess on Derezynski and N3-RSS-500), The performance is lower on OKE15 and OKE16 for the same reason pointed out by De Cao et al. (202T): these datasets annotated with
5

---

Published as a conference paper at ICLR 2022
Table I: InKB Micro F the in-domain and out-of-domain test sets the GERBIL benchmarking platform. For each dataset, bold indicates the best model and underline indicates the second best.
In-domain
Out-of-domain
Method
AIDA
MSNBC
Der
K
R128
R500
OKE15
OKE16
Avg
Hoffart et al. (2011
72.8
65.1
32.6
55.4
46.4
42.4
63.1
0.0
47.2
Steinmetz & Sack
2013
42.3
30.9
26.5
46.8
18.1
20.5
46.2
46.4
34.7
Moro et al. (2014)
48.5
39.7
29.8
55.9
23.0
29.1
41.9
37.7
38.2
Kolitsas 2018
82.4
72.4
34.1
35.2
50.3
38.2
61.9
52.7
53,4
Broscheit 2019)
79.3
Martins al.
2019
81.9
van Hulst et al.
2020
80.5
72.4
41.1
50.7
49.9
35.0
63.1
58.3
56.4
De Cao et
(2021)
83.7
73.7
54.1
60.7
46.7
40.3
56.1
50.0
58.2
EntQA
85.8
72.1
52.9
64.5
54.1
41.9
61.1
51.3
60.5
coreference (i.e., they contain pronouns and common nouns linked to entities) which our model is not trained for, while many other systems have component in their pipelines to handle these cases. We hypothesize that the performance on MSNBC is lagging because it has long documents (544 words per document on average) which are processed in relatively short passages under EntQA due to our computational constraints. Overall, EntQA achieves the best macro-averaged F₁ score across the 8 evaluation datasets (+2.3).
The inference runtime of EntQA is clearly linear in the number of candidate entities K. To get sense of speed, we compared the runtime of EntQA with that of GENRE on the AIDA validation set using GPU on the same machine. GENRE took 1 hour and 10 minutes, excluding 31 minutes first build a prefix tree. EntQA took 20 minutes with minutes with K = 20, excluding hour to first index entity embeddings, yielding F1 scores 87.3, 87.4, and 87.0. Interestingly, we can obtain a significant speedup at a minor cost in performance by decreasing K. We believe this can be useful feature of the model in controlling the speed- performance tradeoff.
100, 10 minutes with K 50, and
We note that there is issue of using different editions of Wikipedia between the systems. For instance, Hoffart et al. 201T)an use the 2010 dump, van Hulst al. (2020) and we use the 2019 dump, whereas Kolitsas al. (2018) and De Cao et al. (2021) the 2014 dump (even though the latter use the 2019 dump for pretraining). Thus there is a concern that differences in performance are due different snapshots of Wikipedia. While address this concern, we find that using different editions of Wikipedia does not fundamentally change the performance of EnIQA, which is consistent with GERBIL's intent of being KB-agnostic. For instance, we obtained the same validation F, on AIDA with our model trained on either the 2014 or 2019 dump. We the KILT edition of Wikipedia mainly for convenience.
consider it of scope in work to fully
3.2.1 OTHER PRACTICAL HIGHLIGHTS
No dictionary. EntQA has no dependence on a mention-candidates dictionary. All previous works rely on a dictionary D V + P(E) that maps mention string to small set of candidate entities
e É E associated with empirical conditional probabilities Pe > 0 (Hoffart et al. 2011 inter alia).
For instance, is essential component of the search procedure in the end-to-end model Kolitsas
ctt al. 2018). While not mentioned in the paper or on the GitHub repository, GENRE <De Cao
2021) also uses the dictionary from Kolitsas et al. (2018) in their prefix tree constrain the beam search (personal communication with one of the authors the paper). Constructing such a
al.
dictionary typically assumes the existence of large KB-specific labeled corpus (e.g., internal links in Wikipedia). EntQA is thus more broadly applicable to KBs without such resources (e.g., for small domain-specific KBs).
No model-specific pretraining. EntQA does require model-specific pretraining; it only uses standard pretrained Transformers for initialization and is directly finetuned on AIDA. This in contrast with GENRE which requires industry-scale pretraining by weak supervision. Specifically,

---

Published as a conference paper at ICLR 2022
Table 2: Ablation study for the retriever module. Each line makes single change from the baseline retriever used in T
We also compare with BM25.
Retriever
Val R@ 100
Baseline
98.2
Omit excluding other gold entities the normalization term NCE
82.7
Train by optimizing the marginal log-likelihood
83.8
Initialize with BERT-laare
94.4
Omit hard negatives in NCE (i.c., negative examples are all random)
94.4
Omit the document-lewe information #1 in the passage representation
96.6
BM25
36.6
Table 3: Ablation study for the reader module. Each line makes single change from the baseline reader used in Table Candidate entities are obtained from the baseline retriever in Table (except the oracle experiment).
Reader
Val
Baseline
87.3
Initialize with BERT-large
85.6
Train by optimizing the marginal log likelihood
86.9
Initialize with ELECTRA large (not finetuned on SQuAD 2.0)
88.4
Omit the reranking probabilities Prerank (i.c., only use span probabilities)
87.9
Omit the document-level information æı in the input passage representation
87.5
Oracle experiment: gold entities the only candidate entities
94.9
GENRE is trained by finetuning BART (Lewis al. 2020) autoregressive EL training examples constructed from all Wikipedia abstract sections on 64 GPUs for 30 hours, followed by finetuning AIDA. Thus training GENRE from seratch is beyond the means of most academic researchers, making it difficult to make substantial changes to the model. EntQA can be trained with academic resources and outperforms GENRE.
3.3 ABLATION STUDIES
The final form of EntQA in Section 3.2 the result of empirically exploring various modeling and optimization choices during development. We present an ablation study to illustrate the impact of these choices.
Retriever Table 2 shows an ablation study for the retriever module. We report top- recall (R@ 100) on the validation set of AIDA. The baseline retriever is initialized with BLINK (Wu al. 2020a), uses the passage representation [CLS] ] [SEP] æ1, and is trained by optimizing the multi- label variant of NCE (1) that considers ization term. We see that the baseline retriever has an extremely high recall (98.2), confirming our hypothesis that is possible accurately infer relevant entities in passage without knowing where they are mentioned. We also see that it is very important to use the proposed multi-label variant of NCE instead of naive NCE that normalizes over all gold entities, which results massive de- crease in recall (82.7). 7). We consider optimizing the marginal log- likeliood (i.c., the log of the sum of the probabilities of gold entities, rather than the sum of the log). but it yields much worse per- formance (83.8). It is helpful to initialize with BLINK rather than BERT-large, use hard negatives NCE, and append I1 to input passages. Table 2 additionally shows the BM25 recall, which is quite poor (36.6). Upon inspection, we find that BM25 fails retrieve diverse entities. For instance, a passage on cricket may have diverse gold entities such as an organization (Lei cesterslll hire County Cricket Club), location (London), and person (Phil Simmonss), but the top enti- ties under BM25 are dominated by person entities (Alan Shipman, Dominique Lewis, etc.). This shows the necessity of explicitly training retriever prioritize diversity in problem.
gold entity at time by excluding others in the normal-

---

Published as a conference paper at ICLR 2022
Table 4: Categorizing errors the validation set passages. The number of passages in each category is given in parentheses. G refers to the gold annotation; P refers to the predicted annotation.
Error Over
Examples (text snippets)
G: england fast bowler [martin mccague] MeCague
(Fill in missing mentions)
(443)
[england erick
cricket team fast bowler [martin mccague Mantin McCague
G: duran, 45, takes little known [mexican]Mesin
[duran] Durin, 45, takes on little known [mexican IMtxico
Under (474)
G: second innings before [simmons Jehi Simmons stepped in
(Bad threshold)
P: second innings hefore simmons stepped in
G: [ato boldon]aw Buldon 1pr [trinidad] Trmided - rpr 20.
P: [ato boldon JAt Boldon 1pr trinidad rpr 20.
Neither (378)
match against yorkshire
[headingleyyJtndngg
(Ambiguous entity)
P: match against yorkshire at (headingley| liteudingly Stadium
G: at the [oval]mee Oval, surrey captain chris lewis
(Ambiguous span)
P: [the oval] Oval, surrey captain chris lewis
G: scores in (englishJengted county championship matches
(Others)
P: scores in [english county championshipl Couny Championship matches
Reader Table shows an ablation study for the reader module. We report F on the validation of AIDA. The baseline reader initialized with ELECTRA-large (Clark et al.) 2019) finetuned on SQuAD 2.0, uses the joint passage-entity input representation [CLS] @ I [SEP] @ + (desc (e) [SEP]. and is trained by optimizing 2 Candidate entities are obtained from the baseline We see that BERT is less performant than ELECTRA for reader initialization,
retriever in Table 2 consistent with findings in the QA literature (Yamada al., 2021). Training by optimizing the marginal log-likelihood is comparable to (2). Interestingly, we find that we can fit the reader just as well without using a SQuAD-finetunet ELECTRA, ranking probabilities, or x1 in passages. However, in preliminary investigation we found that these variants generalized slightly worse outside the training domain, thus we kept our original choice. We discuss other choices of document- conduct oracle experiment in which provide
level information in Appendix B Lastly, only gold entities as candidates to the reader. In this scenario, the reader is very accurate (94.9 F1), suggesting that the main performance bottleneck is correctly distinguishing gold vs non-gold entities from the candidates. We investigate this issue more in depth in the next section.
3.4 ERROR ANALYSIS
To better understand the source of errors made by EnIQA, we examine passages in the validation set for which the model's prediction not completely correct. We partition them into three types: (1) over- vver-predicting (i.e., the gold mentions predicting (i.c., the predicted mentions are a strict subset of the gold mentions), and (3) neither over- under--re Table 4 shows examples of each error type. We find that over-predicting of- ten happens because the model correctly "fills in" entity mentions missing in the gold annotation. Under-predicet predictpe happens most likely because the threshold value is too large catch certain men- tions. Finally, many errors that are neither over- nor under-predicting are largely due to annotation noise, For instance, the predicted entity Headingly Stadi um is correct and more specific en- tity for the span "headingley" than the gold entity Headingly y (a suburb); the predicted span "the oval" is more suitable, or at least as correct as, the gold span "oval" for the entity The Oval.
strict subset of the predicted mentions), (2) under-
We also consider distinguishing MD errors from ED errors on the validation set. EntQA obtains 87.5 overall Fj. When we only measure the correctness of mention spans (equivalent to treating all entity predictions as correct), we obtain 92.3 Fj. When we only measure the correctness of rejecting or accepting candidate entities, we obtain 64.5 F, at the passage level and 89.3 F2 at the document level consider the set of candidates from all passages). The reader's 's relatively low passage-level F1 in rejecting or accepting candidates is consistent with the the oracle experiment in T That is, the main performance bottleneck of EntQA is discriminating gold vs non-gold entities from the candidates, though this should be taken with a grain of salt given the noise in annotation illustrated in Table
8

---

Published as a conference paper at ICLR 2022
4 RELATED WORK
Our work follows the recent trend of formulating language tasks as QA problems, but our knowl- edge we are the first to propose reduction to inverted gpen-domain QA. Most previous works supply questions as input to the system, along with passages in which answer spans are found. They differ only in question formulation, for instance predicate in semantic role labeling, (He et al. 20155) relation type along with its first argument
KB completion (Levy et al., 2017 107,55 [Li et al. 2019), 9 an
entity category in (nested) NER (Li et al., 2020), an auxiliary verb or wh-expression in ellipsis resolution (Aralikatte et al. 2021), and other task-specific questions (McCann et al. 2018). con-
trast, we solve question formulation part of the problem by exploiting recent advances in dense text retrieval.
A notable exception is CorefQA (Wu et al. 2020b). from which we take direct inspiration. In this approach, the authors formulate coreference resolution as QA in which questions are coreferring spans and answers are the spans' antecedents (i.e., earlier spans that belong to the same coreference cluster). Since coreferring spans are unknown, the authors rely on the end-to-end coreference reso- lution model Lef et al. (2017) that produces candidate spans by beam search. In contrast, EntQA handles varying numbers of questions in a simpler framework of text retrieval.
As in this work, some previous works propose methods to handle varying numbers of answer spans for a given question. But their methods are based on one-vs-all classification (i.e., each label is associated with a token-level binary classifier) or reduction tagging (i.e., spans are expressed as BIO-label sequence) (Wu et al. 2020b Li et al. 2019 2020 We found these methods to be ineffective in preliminary experiments, and instead develop more effective inference scheme in which the model delays its final prediction to the end for global reranking (Section 2.3
We discuss pros and cons of EntQA vs other models in practice. While EntQA outperforms GENRE without large-scale weakly supervised pretraining, it involves dense retrieval which incurs large memory footprint to store and index dense embeddings as pointed out by De Cao al. (202T). But can be done single machine with ample RAM (ours has 252G) which is cheap. Bypassing dense retrieval is a unique strength of the autoregressive approach of GENRE and orthogonal to ours; we leave combining their strengths as future work. Our model requires a threshold y for inference, but we find that it is easy to pick a good threshold; we also argue that can be a useful feature in a real-world setting in which the practitioner often needs a customized trade-off between precision and recall. The seeesoldd-bsse inference implies another unique feature of EntQA not explored in this work: it can naturally handle nested entity mentions. We leave nested linking as future work.
byDe
CONCLUSIONS
Existing methods for entity linking suffer from the dilemma of having to predict mentions with- out knowing the corresponding entities. We have presented EntQA, new model that solves this dilemma by predicting entities first and then finding their mentions. Our approach is based on novel reduction to inverse open-domain QA in which we retrieve an unknown number of questions (candidate entities) and predict potentially multiple answer spans (mentions) for each question. Our solution is a simple pipeline that takes full advantage of progress in text retrieval and reading com- prehension. EntQA achieves new state-of-the-ard results the GERBIL benchmarking platform without relying on a KB-specific mention mention-candidates dictionary or expensive model-specific pre- training.
ACKNOWLEDGMENTS
This work was supported by the Google Faculty Research Awards Program.
REFERENCES
Alan Akbik, Duncan Blythe, and Roland Vollgraf. Contextual string embeddings for sequence labeling. In Proceedings of the 27th international conference on computational linguistics, pp- 1638-1649, 2018.

---

Published as a conference paper at ICLR 2022
Chris Alberti, Kenton Lee, and Michael Collins. A bert baseline for the natural questions. arXiv preprint urXiv:1911006644 2019.
Rahul Aralikatte, Matthew Lamm, Daniel Hardt, and Anders Søgaard. Ellipsis resolution question answering: An evaluation. In Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume, pp. 810-817, 2021.
Krisztian Balog, Heri Ramampiaro, Naimdjon Takhirov, and Kjetil Nørvåg. Multi-step classification approaches cumulative citation recommendation. In Proceedings of the 10th conference on open research areas in information retrieval, pp. 121-128, 2013.
Samuel Broscheit. Investigating entity knowledge bert with simple neural end-to-end entity linking. In Proceedings the 23rd Conference Computational Natural Language Learning (CoNLL), pp. 677-685, 2019.
Kevin Clark, Minh- Thang Luong, Quoe V Le, and Christopher Manning. Electra: Pre-training text encoders as discriminators rather than generators. In International Conference on Learning Representations, 2019.
Nicola De Cao, Gautier Izacard, Sebastian Riedel, and Fabio Petroni. Autoregressive entity
retrieval. In International Conference on Learning Representations, 2021. URL https:
//openzeview. http://wwww...tttoooooooooo-aaaa
Leon Derczynski, Diana Maynard, Giuseppe Rizzo, Marieke Van Erp, Genevieve Gorrell, Raphaël Troncy, Johann Petrak, and Kalina Bontcheva. Analysis named entity recognition and linking for tweets. Information Processing & Management, 51(2):32-45 2015.
Jacob Devlin, Ming-Wes Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume I (Long and Short Papers), pp. 4171-41866 2019.
David A Ferrucci. Introduction to this is watson. IBM Journal of Research and Development, 56 (3.4):1-1, 2012.
Octavian- -uggnn Ganea and Thomas Hofmann. Deep joint entity disambiguation with local neural attention. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pp. 2619-2629, 2017.
Daniel Gillick, Sayali Kulkarni, Larry Lansing, Alessandro Presta, Jason Baldridge, Eugene Ie, and Diego Garcia-Olano. Learning dense representations for entity retrieval. In Proceedings of the 23rd Conference on Computational Natural Language Learning (CoNLL), pp. 528-537, Hong Kong, China, November 2019. Association for Computational Linguisties. doi: 10.18653/v1/
K19-1049, URL https:/p//// www. ac wwwwww....c.ooccccco....
www.aaclwwbb.crrr//tttttoooooyee wtt.o////t.tooo///o/oooooo.ee..e 1049
Nitish Gupta, Sameer Singh, and Dan Roth. Entity linking via joint encoding of types, descriptions, and context. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language
Processing, pp. 2681-2690, 2017.
Faegheh Hasibi, Krisztian Balog, and Svein Erik Bratsberg. Exploiting entity linking in queries for entity retrieval. In Proceedings of the 2016 aem international conference on the theory of information retrieval, pp. 209-218. 2016.
Luheng He, Mike Lewis, and Luke Zettlemoyer. Question-answer driven semantic role labeling: Using natural language to annotate natural language. In Proceedings of the 2015 conference on empirical methods in natural language processing, pp. 643-653, 2015.
Johannes Hoffart, Mohamed Amir Yosef, Iariaa Bordino, Hagen Fürstenau, Manfred Pinkal, Marc Spaniol, Bilyana Taneva, Stefan Thater, and Gerhard Weikum. Robust disambiguation of named entities in text. In Proceedings of the 2011 Conference on Empirical Methods in Natural Lan- guage Processing, pp. 7922792,10 2011.
10

---

Published as a conference paper at ICLR 2022
Johannes Hoffart, Stephan Seufert, Dat Ba Nguyen, Martin Theobald, and Gerhard Weikum. Kore: keyphrase overlap relatedness for entity disambiguation. In Proceedings of the 2 ACM inter- national conference Information and knowledge management, pp- 545-554, 2012.
Jeff Johnson, Matthijs Douze, and Hervé Jégou. Billion-scale similarity search with gpus. IEEE Transactions on Big Data, 2019.
Vladimir Karpukhin, Barlas Oguz, Sewon Min, Patrick Lewis, Ledell Wu, Sergey Edunov, Danqi Chen, and Wen-tau Yih. Dense passage retrieval for open-domain question answering. In Proceedings of the 2020 Conference Empirical Methods in Natural Language Processing (EMNLP), pp. 6769-6781, 2020.
Diederik Kingma and Jimmy Ba. Adam: A method for stochastic optimization. ICLR (Poster), 2015.
Nikolaos Kolitsas, Octavian-Fugen Ganea, and Thomas Hofmann. End-to-end neural entity linking. In Proceedings of the 22nd Conference on Computational Natural Language Learning, pp. 519- 529, 2018.
Kenton Lee, Luheng He, Mike Lewis, and Luke Zettlemoyer. End-to-end neural coreference res- olution. In Proceedings of the 2017 Conference Empirical Methods in Natural Language Processing, pp. 188-197,201 2017.
Omer Levy, Minjoon Seo, Eunsol Choi, and Luke Zettlemoye. Zero-shot relation extraction via reading comprehension. In Proceedings of the 21 Conference on Computational Natural Lan- guage Learning (CoNLL 2017). pp- 333-342, 2017.
Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Veselin Stoyanov, and Luke Zettlemoyer. Bart: Denoising sequence-to-sequeee pre- training for natural language generation, translation, and comprehension. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pp. 7871-7880, 2020.
Xiaoya Li, Fan Yin, Zijun Sun, Xiayu Li, Arianna Yuan, Duo Chai, Mingxin Zhou, and Jiwei Li. Entity-relation extraction as multi-turn question answering. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pp. 1340-1350, 2019.
Xiaoya Li, Jingrong Feng, Yuxian Meng, Qinghong Han, Fei Wu, and Jiwei Li. A unified mre frame- work for named entity recognition. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics. pp. 5849-5888 5859, 2020.
Xiao Ling, Sameer Singh, and Daniel S Weld. Design challenges for entity linking. Transactions of the Association for Computational Linguistics, 3:315-328, 2015.
Pedro Henrique Martins, Zita Marinho, and André FT Martins. Joint learning of named entity recognition and entity linking. In Proceedings of the 57th Anmual Meeting of the Association for Computational Linguistics: Student Research Workshop, pp. 190-196. 2019.
Bryan McCann, Nitish Shirish Keskar, Caiming Xiong, and Richard Socher. The natural language decathlon: Multitask learning as question answering. arXiv preprint arXiv: 1806.88077777 08730, 2018.
Andrea Moro, Alessandro Raganato, and Roberto Navigli. Entity linking meets word sense disam- biguation: unified approach. Transactions of the Association for Computational Linguistics, 2: 231-244 2014.
Andrea Giovanni Nuzzolese, Anna Lisa Gentile, Valentina Presutti, Aldo Gangemi, Darío Garigliotti, and Roberto Navigli. Open knowledge extraction challenge. In Semantic Web Evalu- ation Challenges, pp. 3-15. Springer, 2015.
Fabio Petroni, Aleksandra Piktus, Angela Fan, Patrick Lewis, Majid Yazdani, Nicola De Cao, James Thorne, Yacine Jernite, Vladimir Karpukhin, Jean Maillard, Vassilis Plachouras, Tim Rocktäschel, and Sebastian Riedel. KILT: a benchmark for knowledge intensive language tasks. In Proceedings of the 2021 Conference of the North American Chapter of the Associ- ation for Computational Linguistics: Human Language Technologies, pp. 5523-2544, Online, June 2021. Association for Computational Linguistics. doi: 10.18653/v 1/202 mamacl-main.... URL https: //ac antho ogy. org/202r//21 nwac - 200

---

Published as a conference paper at ICLR 2022
Table 5: GERBIL test scores with and without using the first document token as document-level topical information.
Topic Info
AIDA
MSNBC
Der
K50
R128
R500
OKE15
OKE16
Avg
None
81.7
72.2
52.5
64.2
54.0
40.3
59.0
48.9
59.1
First token
85.8
72.1
52,9
64.5
54.1
41.9
61.1
51.3
60.5
Pranav Rajpurkar, Robin Jia, and Perey Liang. Know what you dont know: Unanswerable questions for squad. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pp. 784-789, 2018.
Ridho Reinanda, Edgar Meij, and Maarten de Rijke. Mining, ranking and recommending entity aspects. In Proceedings of the 38th International ACM SIGIR Conference Research and De- velopment in Information Retrieval, pp. 263-272, 2015.
Michael Röder, Ricardo Usbeck, Sebastian Hellmann, Daniel Gerber, and Andreas Both. N collection of datasets for named entity recognition and disambiguation in the nip interchange format. In LREC, pp. 3529-3533 2014.
Michael Röder, Ricardo Usbeck, and Axel-Cyrille Ngonga Ngomo. Cerbil-benchmarking named entity recognition and linking consistently. Semantic Web, 95):6055 2018.
Bill Slawski. How google uses named entity disambiguation for entities with the httpy:/e/ww..eorteeeeeeeeeeee www 2015/90 disambiquate entities in queries and paciddpaaes/// Accessed: 2021-09-27.
same names, September 2015. URL httpt::////wtt wwwbyt
Nadine Steinmetz and Harald Sack. Semantic multimedia information retrieval based on contextual descriptions. Extended Semantic Web Conference, pp. 382-396. Springer, 2013.
Johannes M van Hulst, Faegheh Hasibi, Koen Dercksen, Krisztian Balog, and Arjen P de Vries. Rel: An entity linker standing on the shoulders of giants. In Proceedings of the 43rd International ACM SIGIR Conference on Research and Development in Information Retrieval, pp. 2197-2200, 2020.
Ledell Wu, Fabio Petroni, Martin Josifoski, Sebastian Riedel, and Luke Zettlemoyer. Scalable zero- shot entity linking with dense entity retrieval. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pp. 6397-6407, 2020a.
Wei Wu, Fei Wang, Arianna Yuan, Fei Wu, and Jiwei Li. Corefqa: Coreference resolution as query-based span prediction. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pp. 6953-6963, 2020b.
Chenyan Xiong, Jamie Callan, and Tie- Yan Liu. Word-entity duet representations for document ranking. In Proceedings of the 40th International ACM SIGIR conference on research and devel- opment in information retrieval, pp. 763-772, 2017.
Ikuya Yamada, Akari Asai, and Hannanch Hajishirzi. Efficient passage retrieval with hashing for open-domain question answering. arXiv preprint arXiv:2106.00888 2021.
Yi Yang, Ozan Irsoy, and Kazi Shefaet Rahman. Collective entity disambiguation with structured gradient tree boosting. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers), pp. 777-786, 2018
Wenzheng Zhang and Karl Stratos. Understanding hard negatives in noise contrastive estimation. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pp. 1090-1101, 2021.
12

---

Published as a conference paper at ICLR 2022
A THRESHOLD OPTIMIZATION
To see if possible to improve over the static threshold value 7 0.05, we tried automatically calibrating 7 based on the AIDA validation performance by considering every effective threshold obtained from a sorted list of probabilities of labeled mentions. The best threshold was = 0.03 840 The validation F1 score improved from 87.32 to 87.75, and the GERBIL test score improved from 60.46 to 60.55. Thus threshold optimization can yield minor improvement, but overall we find that EntQA robust choices of threshold in a reasonable range.
B DOCUMENT-LEVEL INFORMATION
We explored various ways of injecting document-level information in paragraphs. We tried the first token, the first sentence, and continuous topic embedding (obtained by averaging all token embed- dings the document). We settled on the first-token version because it gave the best performance. For many of the GERBIL datasets, however, obtain almost the same performance with without the topic information. As it is somewhat dataset-specific (e.g., the first word in AIDA is always the topic word), leave it option in our model for the to decide. Thows the GERBIL performance without any topic information vs with the first token in the document.
13


Reviews:
[{"review_id": "7UuS46-EtBn", "reviewer": "Reviewer_SZGo", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strength:\n* The novel entity linking paradigm proposed by this paper is insightful and the performance gain is significant.\n* The proposed paradigm addresses a key issue in prior entity linking works. In EntQA, mention detection happens after entity detection, so that mention detection can utilize entity information. \n* The proposed QA paradigm allows EntQA to take advantage of existing question answering datasets. Pretraining on open domain question answering datasets may improve the generalizability of EntQA. So not surprisingly, EntQA performs better on out-of-domain datasets.\n* The proposed method has several practical advantages. It costs less GPU hours and memory space in comparison with prior SOTA according to the paper. However, providing the exact number will be much better.\n\nWeakness: \n* When discussing the difference between entity linking paradigms, this paper doesn’t compare with GENRE [1], although it is listed as a baseline. As the main contribution of this paper is the novel paradigm, I would like to see a section instead of a brief paragraph in introduction to discuss the theoretical and empirical difference between different paradigms. However, considering that Table 1 already shows the proposed paradigm performs better than other paradigms empirically, the theoretical analysis can be left for future work.\n* The term ‘unknown entities’ is confusing, because it can also refer to entities not covered by KB. I suggest the authors clarify the definition. I think it is better to say ‘predict mentions when not knowing the corresponding entities’ than ‘predict mentions of unknown entities’. Besides, I wonder how mentions of entities not covered by KB will influence different paradigms, although it may be a minor issue. \n* The effectiveness of EntQA is only shown by overall performance. It would be better to have experiments to compare the mention detection results of different methods.\n\n[1] De Cao, N., Izacard, G., Riedel, S., & Petroni, F. (2020, September). Autoregressive Entity Retrieval. In International Conference on Learning Representations.\n", "summary_of_the_paper": "This paper proposes a new paradigm for entity linking. The proposed method EntQA retrieves candidate entities first, and then uses candidate entities as queries to find mention spans. \nEntQA addresses the issue of finding mentions without knowing their entities and offers many practical advantages. Experimental results show that EntQA can achieve good performance without mention-candidates dictionaries and large-scale task-specific weak supervision.\n", "summary_of_the_review": "The insights and novel paradigm for entity linking proposed by this paper are attractive. Although the experiments can be more sufficient, the key claims are supported by the experimental results. I recommend accepting this paper.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "6h2rzWTwkHW", "reviewer": "Reviewer_97R7", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "This paper addresses a problem of entity linking. The authors propose a reformulation of the problem as an inverted open-domain Question-Answers (QA) The authors used dense retriever in the first phase of their algorithm (retrieving top K entities) and then in the second phase authors trained reader mouse for ranking and extracting entity mentions from the output of the retriever.\nThe main advantages of the proposed method (besides outperforming previous strategies) is the data efficiency gained by eliminating dependency on hardcoded mention-candidate dictionary. Instead it finds entities from text passages and then finds lined mentions in test passages.\nAuthors work comes with strong backing from trying they model on GERBIL benchmark platform (trying it on various datasets) I especially appreciated the authors explaining the reason why their method did not do as well on OKE15/16 datasets compared to how it performed on other datasets.\nRegarding the reproducibility of the results, the authors did not share their source code so it might take some effort to reproduce what they have done. On the other hand, they provided very detailed description of their training process including very helpful Model details section with lot's of relevant details parameters used with training and inference.\nAuthors provided very insightful discussion of their results. In addition to what's been provided I would be interested to learn how quick is the inference and how the time to run inference is impacted by the choice of K.", "summary_of_the_paper": "This paper improves on the stat-of-the-art in entity linking.  Entity linking involves connecting entities to their mentions in test passages.\nCurrent solutions rely on finding mentions first before running entity disambiguation stage which is difficult because mentions are being searched for unknown entities in the first stage.\nThe authors address this challenge by rearranging the standard QA pipeline by learning questions first (finding entities) and searching for answers (mentions) at a later stage.\nThis approach appears to be well-researched. The authors tested their model on GERBIL benchmarking platform and achived very positive results.\n", "summary_of_the_review": "Overall, I believe this paper is a worthy addition to this year's conference. It tackles an important problem and provides a better performing, simpler and more intuitive solution compare to previous work.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "6BcIANkl6i3", "reviewer": "Reviewer_Uses", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "This paper is clearly written, has a working original idea, and provides a great new option for entity linking that has empirical promise. It's in good shape, and I don't have a lot of criticisms. Just a comments and questions below:\n\nPage 1 - \"End-to-end models alleviate the problem of error propagation, but the search is only approximate and the dilemma, albeit to a lesser degree, remains\" - I think this sentence need some additional justification. Sure, search is approximate. That on its own doesn't mean that the dilemma remains. How would you establish that the dilemma remains? \n\nPage 3 - \"In our experiments we simply set ψtopic(x) = x1 ∈V (i.e., the first token in the document).\" That seems odd. The first token may not be very useful. It could be \"The\". Could you use the encoded first phrase?\n\nPage 3 - \"We write ⊕to denote the text concatenation (we insert a special symbol to represent the concatenation)\" I don't understand the parenthetical. Can you elaborate?\n\nThe bad threshold suggest that the model is perhaps not calibrated in some key way. Could an explicit calibration step on the validation set help fix these errors? More broadly, the fixed threshold seems like the weakest part of the system. What ideas do you have about making it more flexible or adaptable? Did you try other approaches to thresholding and found them to not work? These findings would be good to include in the paper.", "summary_of_the_paper": "Standard entity linking first does mention detection and then the actual linking of mentions to entities in the knowledge base, and the authors propose to flip this by using open domain QA. They first use the text to find candidate entities in the knowledge base. Then for each of these candidates, they look for possible mentions in the text. The model performs well on average, and has some nice advantages such as no mention-candidates dictionary and simplicity of generalization to out-of-domain tasks.", "summary_of_the_review": "Solid paper about an original approach to entity linking that performs pretty well.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper casts entity linking in a retrieve-then-read framework by first retrieving entity candidates and then finding their mentions via reading comprehension. All reviewers agree that the proposed approach is novel, well-motivated, and simple yet performant. The authors have done a good job of addressing all the concerns raised, and the reviewers are unanimous in their recommendation for accepting the paper. I hope the authors will also incorporate the feedback and their responses in the final version.
Decision: Accept (Spotlight)