Learning Cross-Context Entity Representations from Text

Language modeling tasks , in which words , or word-pieces , are predicted on the basis of a local context , have been very effective for learning word embeddings and context dependent representations of phrases . Motivated by the observation that efforts to code world knowledge into machine readable knowledge bases or human readable encyclopedias tend to be entity-centric , we investigate the use of a fill-in-the-blank task to learn context independent representations of entities from the text contexts in which those entities were mentioned . We show that large scale training of neural models allows us to learn high quality entity representations , and we demonstrate successful results on four domains : ( 1 ) existing entity-level typing benchmarks , including a 64 % error reduction over previous work on TypeNet ( Murty et al. , 2018 ) ; ( 2 ) a novel few-shot category reconstruction task ; ( 3 ) existing entity linking benchmarks , where we match the state-of-the-art on CoNLLAida without linking-specific features and obtain a score of 89.8 % on TAC-KBP 2010 without using any alias table , external knowledge base or in domain training data and ( 4 ) answering trivia questions , which uniquely identify entities . Our global entity representations encode fine-grained type categories , such as Scottish footballers , and can answer trivia questions such as Who was the last inmate of Spandau jail in Berlin ? . 1 INTRODUCTION . A long term goal of artificial intelligence has been the development and population of an entitycentric representation of human knowledge . Efforts have been made to create the knowledge representation with knowledge engineers ( Lenat et al. , 1986 ) or crowdsourcers ( Bollacker et al. , 2008 ) . However , these methods have relied heavily on human definitions of their ontologies , which are both limited in scope and brittle in nature . Conversely , due to recent advances in deep learning , we can now learn robust general purpose representations of words ( Mikolov et al. , 2013 ) and contextualized phrases ( Peters et al. , 2018 ) directly from large textual corpora . In particular , we observe that existing methods of building contextualized phrase representations capture a significant amount of local semantic context ( Devlin et al. , 2019 ) . We hypothesize that by learning an entity encoder which aggregates all of the textual contexts in which an entity is seen , we should be able to extract and condense general purpose knowledge about that entity . Consider the following contexts in which an entity mention has been replaced a [ MASK ] : . . . the second woman in space , 19 years after [ MASK ] . . . . [ MASK ] , a Russian factory worker , was the first woman in space . . . . . . [ MASK ] , the first woman in space , entered politics . . . . As readers , we understand that first woman in space is a unique identifier , and we are able to fill in the blank unambiguously . The central hypothesis of this paper is that , by matching entities to the contexts in which they are mentioned , we should be able to build a representation for Valentina Tereshkova that encodes the fact that she was the first woman in space , that she was a politician , etc . and that we should be able to use these representations across a wide variety of downstream entity-centric tasks . We present RELIC ( Representations of Entities Learned in Context ) , a table of independent entity embeddings that have been trained to match fixed length vector representations of the textual context in which those entities have been seen . We apply RELIC to entity typing ( mapping each entity to its properties in an external , curated , ontology ) ; entity linking ( identifying which entity is referred to by a textual context ) , and trivia question answering ( retrieving the entity that best answers a question ) . Through these experiments , we show that : • RELIC accurately captures categorical information encoded by human experts in the Freebase and Wikipedia category hierarchies . We demonstrate significant improvements over previous work on established benchmarks , including a 64 % error reduction in the TypeNet low data setting . We also show that given just a few exemplar entities of a given category such as Scottish footballers we can use RELIC to recover the remaining entities of that category with good precision . • Using RELIC for entity linking can match state-of-the-art approaches that make use of non-local and non-linguistic information about entities . On the CoNLL-Aida benchmark , RELIC achieves a 94.9 % accuracy , matching the state-of-the-art of Raiman & Raiman ( 2018 ) , despite not using any entity linking-specific features . On the TAC-KBP 2010 benchmark RELIC achieves 89.8 % accuracy , just behind the top ranked system ( Raiman & Raiman , 2018 ) , which makes use of external knowledge bases , alias tables , and taskspecific hand-engineered features . • RELIC learns better representations of entity properties if it is trained to match just the contexts in which entities are mentioned , and not the surface form of the mention itself . For entity linking , the opposite is true . • We can treat the RELIC embedding matrix as a store of knowledge , and retrieve answers to questions through nearest neighbor search . We show that this approach correctly answers 51 % of the questions in the TriviaQA reading comprehension task ( Joshi et al. , 2017 ) despite not using the task ’ s evidence text at inference time . The questions answered correctly by RELIC are surprisingly complex , such as Who was the last inmate of Spandau jail in Berlin ? 2 RELATED WORK . Entity linking The most widely studied entity-level task is entity linking—mapping each entity mention onto a unique entity identifier . The Wikification task ( Ratinov et al. , 2011 ; Cheng & Roth , 2013 ) , in particular , is similar to the work presented in this paper , as it requires systems to map mentions to the Wikipedia pages describing the entities mentioned . There is significant previous work that makes use of neural context and entity encoders in downstream entity linking systems ( Sun et al. , 2015 ; Yamada et al. , 2016 ; 2017 ; Gupta et al. , 2017 ; Murty et al. , 2018 ; Kolitsas et al. , 2018 ) , but that previous work focuses solely on discriminating between entities that match a given mention according to an external alias table . Here we go further in investigating the degree to which RELIC can capture world knowledge about entities . Mention-level entity typing Another well studied task is mention-level entity typing ( e.g . Ling & Weld , 2012 ; Choi et al. , 2018 ) . In this task , entities are labeled with types that are supported by the immediate textual context . For example , given the sentence ‘ Michelle Obama attended her book signing ’ , Michelle Obama should be assigned the type author but not lawyer . Subsequently , mention-level entity typing systems make use of contextualized representations of the entity mention , rather than the global entity representations that we focus on here . Entity-level typing An alternative notion of entity typing is entity-level typing , where each entity should be associated with all of the types supported by a corpus . Yaghoobzadeh & Schütze ( 2015 ) and Murty et al . ( 2018 ) introduce entity-level typing tasks , which we describe more in Section 5.2 . Entity-level typing is an important task in information extraction , since most common ontologies make use of entity type systems . Such tasks provide a strong method of evaluating learned global representations of entities . Using knowledge bases There has been a strong line of work in learning representations of entities by building knowledge base embeddings ( Bordes et al. , 2011 ; Socher et al. , 2013 ; Yang et al. , 2014 ; Toutanova et al. , 2016 ; Vilnis et al. , 2018 ) , and by jointly embedding knowledge bases and information from textual mentions ( Riedel et al. , 2013 ; Toutanova et al. , 2015 ; Hu et al. , 2015 ) . Das et al . ( 2017 ) extended this work to the SPADES fill-in-the-blank task ( Bisk et al. , 2016 ) , which is a close counterpart to RELIC ’ s training setup . However , we note that all examples in SPADES correspond to a fully connected sub-graph in Freebase Bollacker et al . ( 2008 ) . Subsequently , the contents are very limited in domain and Das et al . ( 2017 ) show that it is essential to use the contents of Freebase to do well on this task . We consider the unconstrained TriviaQA task ( Joshi et al. , 2017 ) , introduced in Section 5.5 , to be a better evaluation for open domain knowledge representations . Fill-in-the-blank tasks There has been significant previous work in using fill-in-the-blank losses to learn context independent word representations ( Mikolov et al. , 2013 ) , and context-dependent word and phrase representations ( Dai & Le , 2015 ; Peters et al. , 2018 ; Radford et al. , 2018 ; Devlin et al. , 2019 ) . Cloze-style tasks , in which a system must choose which of a few entities best fill a blanked out span , have also been proposed as a method of evaluating reading comprehension ( Hermann et al. , 2015 ; Long et al. , 2016 ; Onishi et al. , 2016 ) . For entities , Long et al . ( 2017 ) consider a similar fill-in-the-blank task as ours , which they frame as rare entity prediction . Yamada et al . ( 2016 ) and Yamada et al . ( 2017 ) train entity representations using a fill-in-the-blank style loss and a bag-of-words representation of mention contexts . Yamada et al . ( 2016 ; 2017 ) in particular take an approach that is very similar in motivation to RELIC , but which focuses on learning entity representations for use as features in downstream classifiers that model non-linear interactions between a small number of candidate entities . In Section 5.4 , we show that Yamada et al . ( 2017 ) ’ s entity embeddings are good at capturing broad entity types such as Tennis Player but less good at capturing more complex compound types such as Scottish Footballers . In Section 5.1 , we also show that by performing nearest neighbor search over the 818k entities in the TAC knowledge base , RELIC can surpass Yamada et al . 2017 ’ s performance on the TAC-KBP 2010 entity linking benchmark ( Ji et al. , 2010 ) . This is despite the fact that Yamada et al . massively restrict the linking search space with an externally defined alias table , and incorporate task-specific hand-engineered features . On the CoNLL-Aida benchmark , we show that RELIC surpasses Yamada et al . 2017 and matches Raiman & Raiman 2018 without using any entity linking-specific features . 3 LEARNING FROM CONTEXT . 3.1 RELIC TRAINING INPUT . Let E = { e0 . . . eN } be a predefined set of entities , and let V = { [ MASK ] , [ Es ] , [ Ee ] , w1 . . . wM } be a vocabulary of words . A context x = [ x0 . . . xl ] is a sequence of words xi ∈ V . Each context contains exactly one entity start marker xk = [ Es ] and one entity end marker xj = [ Ee ] , where j − k > 1 . The sequence of words between these markers , [ xk+1 . . . xj−1 ] , is the entity mention . Our training data is a corpus of ( context , entity ) pairs D = [ ( x0 , y0 ) . . . ( xN , yN ) ] . Each yi ∈ E identifies an entity that corresponds to the single entity mention in xi . We train RELIC to correctly match match the entities in D to their mentions . We will experiment with settings where the mentions are unchanged from the original corpus , as well as settings where with some probability m ( the mask rate ) all of the words in the mention have been replaced with the uninformative [ MASK ] symbol . We hypothesize that this parameter will play a role in the effectiveness of learned representations in downstream tasks . For clean training data , we extract our corpus from English Wikipedia1 . See Section 4 for details .

Summary: This paper proposes to learn entity representations by matching entities to the context it occurs in. It also shows that using these representations is very effective for a wide variety of down-stream entity-centric tasks such as entity typing, linking, and answering entity centric trivia questions. They train the model using a corpus of entity linked Wikipedia contexts (sentences unto length 128 tokens). The context is encoded with a BERT model and the CLS representation is used as the representation of context. After obtaining the representation, they train the entity embedding (present in the sentence) to be similar to the context embedding. They test their embeddings on few down-stream entity-centric tasks — linking, typing and trivia question answering.