EntQA: Entity Linking as Question Answering

1 INTRODUCTION . We consider the most general form of entity linking ( EL ) in which a system , given a document , must both extract entity mentions and link the mentions to their corresponding entries in a knowledge base ( KB ) . EL is a foundational building block in automatic text understanding with applications to question answering ( QA ) ( Ferrucci , 2012 ) , information retrieval ( Xiong et al. , 2017 ; Hasibi et al. , 2016 ; Balog et al. , 2013 ; Reinanda et al. , 2015 ) , and commercial recommendation systems ( Yang et al. , 2018 ; Slawski , 2015 ) . The output space in EL is intractably large . Any subset of all possible spans in the document linked to any KB entries ( typically in the order of millions ) can be a system output . To get around the intractability , existing methods decompose EL into mention detection ( MD ) and entity disambiguation ( ED ) and tackle them with varying degrees of independence . In all cases , however , the order of these two subproblems is MD followed by ED : first the system identifies potential entity mentions , and then the mentions are resolved to KB entries . Previous works either assume that mentions are given ( Gupta et al. , 2017 ) , run an off-the-shelf named-entity recognition ( NER ) system to extract mentions and resolve them by ED ( MD→ED pipeline ) ( Hoffart et al. , 2011 ; Ling et al. , 2015 ; van Hulst et al. , 2020 ) , or train an end-to-end model that jointly performs MD→ED by beam search ( Kolitsas et al. , 2018 ; Cao et al. , 2021 ) . A limitation of performing MD before ED is that it requires finding mentions of unknown entities . By definition , a mention needs an entity ( i.e. , a mention of what ? ) . Existing methods suffer from the dilemma of having to predict mentions before what they refer to , which is unnatural and difficult . For example , the MD→ED pipeline heuristically extracts mentions from spans of named entities found by a third-party NER system , and the performance bottleneck is often errors in MD propagated to ED . End-to-end models alleviate the problem of error propagation , but the search is only approximate and the dilemma , albeit to a lesser degree , remains . In this work , we propose flipping the order of the two subproblems and solving ED before MD . We first find candidate entities that might be mentioned in the given document , then for each candidate find its mentions if possible . Our key observation is that while finding mentions is difficult without the knowledge of relevant entities , finding relevant entities is easy without the knowledge of their specific mentions . This simple change fundamentally solves the dilemma above since identifying mentions of a particular entity is well defined . We cast the problem as inverted open-domain QA . Specifically , given a document , we use a dual encoder retriever to efficiently retrieve top-K candidate entities from the KB as “ questions ” . Then we apply a deep cross-attention reader on the document for each candidate to identify mentions of the candidate in the document as “ answer spans ” . Unlike in standard QA , the model must predict an unknown number of questions and answers . We present a simple and effective solution based on thresholding . We call our model EntQA , standing for Entity linking as Question Answering . Beyond conceptual novelty , EntQA also offers many practical advantages . First , EntQA allows us to piggyback on recent progress in dense entity retrieval and open-domain QA . For instance , we warm start EntQA with the BLINK entity retriever ( Wu et al. , 2020a ) and ELECTRA finetuned on a QA dataset ( Clark et al. , 2019 ) to obtain an easy improvement . Second , EntQA has no dependence on a hardcoded mention-candidates dictionary which is used in previous works to reduce the search space and bias the model ( Ganea & Hofmann , 2017 ; Kolitsas et al. , 2018 ; Cao et al. , 2021 ) . The dictionary is typically constructed using a large KB-specific labeled corpus ( e.g. , Wikipedia hyperlinks ) , thus having no dependence on it makes our approach more broadly applicable to KBs without such resources . Third , training EntQA is data efficient and can be done with an academic budget , in contrast with GENRE ( Cao et al. , 2021 ) which requires industry-scale pretraining by weak supervision . EntQA achieves strong performance on the GERBIL benchmarking platform ( Röder et al. , 2018 ) . The in-domain F1 score on the test portion of the AIDA-CoNLL dataset is 85.8 ( 2.1 absolute improvement ) . The macro-averaged F1 score across 8 evaluation datasets is 60.2 ( 2.0 absolute improvement ) .1 We analyze EntQA and find that its retrieval performance is extremely strong ( over 98 top-100 recall on the validation set of AIDA ) , verifying our hypothesis that finding relevant entities without knowing their mentions is easy . We also find that the reader makes reasonable errors such as accurately predicting missing hyperlinks or linking a mention to a correct entity that is more specific than the gold label . 2 MODEL . Let E denote the set of entities in a KB associated with a text title and description . Let V denote the vocabulary and X = { x ∈ VT : 1 ≤ T ≤ Tmax } the set of all documents up to length Tmax . EL is the task of mapping x ∈ X to y ∈ P ( Y ( x ) ) where Y ( x ) = { ( s , t , e ) : 1 ≤ s ≤ t ≤ |x| , e ∈ E } is the set of all possible linked spans in x and P is the power set . The size of the output space is O ( 2T 2 max|E| ) where |E| is typically very large ( e.g. , around 6 million in Wikipedia ) and Tmax can also be large ( e.g. , > 3000 in AIDA ) , ruling out any naive exhaustive search as a feasible approach . EntQA decomposes EL into two subproblems : entity retrieval and question answering . More specifically , given a document x ∈ X , 1 . The retriever module retrieves top-K candidate entities that might be mentioned in x . 2 . The reader module extracts mentions of each candidate entity in x ( or rejects it ) , then returns a subset of globally reranked labeled mentions as the final prediction . Our approach bears superficial similarities to a standard framework in open-domain QA that pipelines retrieval and span finding ( Karpukhin et al. , 2020 , inter alia ) , but it has the following important differences . First , instead of retrieving passages given a question , it retrieves questions ( i.e. , candidate entities ) given a passage . Second , even when considering a single question , there can be multiple answer spans ( i.e. , mentions ) instead of one . Both the number of gold entities present in a document and the number of mentions of each gold entity are unknown , making this setting more challenging than standard QA in which we only need to find a single answer span for a single question on a passage . Input representation . Both the retriever and the reader work with text representations of documents and entities , thus applicable to a zero-shot setting ( e.g. , linking to a new KB at test time by reading entity descriptions ) . We use the title φtitle ( e ) ∈ V+ and the description φdesc ( e ) ∈ V+ to represent an entity e ∈ E . Since a document x ∈ X is generally too long to encode with a Transformer encoder which has a quadratic dependency on the input length , we break it down in mx ∈ N overlapping passages p1 ( x ) . . . pmx ( x ) ∈ VL of length L with stride S ( e.g. , L = 32 and S = 16 ) and operate at the passage-level similarly as in QA ( Alberti et al. , 2019 ) . When a document is long , 1We will release code to reproduce our experiments upon publication . individual passages may lose global information . For long documents , we find it beneficial to carry a document-level topical text ψtopic ( x ) ∈ V+ across passages in that document ( e.g. , first sentence ) . We emphasize that we do not use any extra information outside the document . In our experiments we simply set ψtopic ( x ) = x1 ∈ V ( i.e. , the first token in the document ) . Notation . We write encθS : VT → Rd×T to denote a Transformer encoder that maps any token sequence to the same-length sequence of corresponding contextual embeddings ; the symbol S is used to distinguish different encoders . We assume the usual special tokens in the input popularized by BERT ( Devlin et al. , 2019 ) : [ CLS ] to represent the whole input and [ SEP ] to indicate an input boundary . We write ⊕ to denote the text concatenation ( we insert a special symbol to represent the concatenation ) . We write Mi ∈ Rd to denote the i-th column of matrix M ∈ Rd×T . 2.1 RETRIEVER . Given a passage p ∈ V+ in document x and an entity e ∈ E , the retriever computes P = encθP ( [ CLS ] p⊕ ψtopic ( x ) [ SEP ] ) Ee = encθE ( [ CLS ] φtitle ( e ) ⊕ φdesc ( e ) [ SEP ] ) scoreθretr ( p , x , e ) = P > 1 E e 1 At inference time , we precompute Ee ∈ Rd for each e ∈ E and use Faiss ( Johnson et al. , 2019 ) for fast top-K retrieval . Training . We train the retriever by a multi-label variant of noise contrastive estimation ( NCE ) . Given a passage p in document x , we have a set of multiple gold entities E ( p ) ⊂ E that are mentioned in the passage and optimize the per-example objective max θ ∑ e∈E ( p ) log ( exp ( scoreθretr ( p , x , e ) ) exp ( scoreθretr ( p , x , e ) ) + ∑ e′∈N ( E , p ) exp ( scoreθretr ( p , x , e ′ ) ) ) ( 1 ) where N ( E , p ) ⊂ E\E ( p ) is a set of negative examples that excludes all gold entities E ( p ) . The objective effectively constructs |E ( p ) | independent NCE instances , each of which treats a gold entity as the only correct answer while ensuring that other gold entities are not included in negative examples . We obtain 90 % of N ( E , p ) by sampling entities uniformly at random from E\E ( p ) and 10 % by hard negative mining ( i.e. , using highest-scoring incorrect entities under the model ) , which is well known to be beneficial in entity retrieval ( Gillick et al. , 2019 ; Wu et al. , 2020a ; Zhang & Stratos , 2021 ) .

Summary: This paper proposes a new paradigm for entity linking. The proposed method EntQA retrieves candidate entities first, and then uses candidate entities as queries to find mention spans.  EntQA addresses the issue of finding mentions without knowing their entities and offers many practical advantages. Experimental results show that EntQA can achieve good performance without mention-candidates dictionaries and large-scale task-specific weak supervision.