Improving Named Entity Disambiguation using Entity Relatedness within Wikipedia
By: Will Seaton
Tags: Editors Pick, Kensho, Named Entity Recognition, NLP

Introduction

Named Entity Disambiguation (NED) is a research area of Natural Language Processing (NLP) focused on linking a reference within a unit of text to its corresponding entity in some knowledge base, such as a node in a knowledge graph. Accurate NED is critical for modern technology companies to process and link documents, reports, press releases and other written material with meaningful contextual information to entries in knowledge bases, in order to improve understanding of an entity for research and commercial purposes.

While traditional NLP approaches to the problem have focused on early stages of such data pipelines — like Named Entity Recognition, string normalization and context windows — recent research has started to focus on comparing named entities themselves within a contextual window — such as a single sentence — to produce better link predictions. Identifying the strength of relationship between named entities within a contextual window involves calculating what we refer to as Named Entity Congruence, or the contextual and semantic relatedness of named entities. In partnership with Kensho Technologies, we develop several methodologies for incorporating congruence in NED using open-source Wikimedia data, and demonstrate its potential to improve performance on entity disambiguation tasks.

Kensho Technologies

Kensho is a leading AI and ML company and part of S&P Global since being acquired in 2018 in the largest AI acquisition in history to date. One of their primary offerings is Kensho Link, a product that allows quick ingestion of a company’s data and accurate linking to S&P Global’s Company IDs. This allows companies to rapidly build a unique and powerful knowledge base by combining their own data with S&P’s world class research data. Their expertise and long partnership with Harvard University’s Institute for Applied Computational Science helped guide the formation of this research question.

Named Entity Recognition

To better understand the goal of Named Entity Disambiguation, it is helpful to place it as the last step in a longer NLP pipeline that begins with Named Entity Recognition. Named Entity Recognition identifies within a unit of text the word or words that represent an entity or object — such as names, organizations, locations, or other proper nouns.

Highlighted words are example of entities identified through Named Entity Recognition.

Once a named entity has been identified, it must be matched to a corresponding node in a knowledge base in order to grow that knowledge base with primary source documentation and improve understanding of the node. The simplest way to link a named entity is by direct string matching. If the named entity is “Harvard University,” it is highly probable it is referencing the node “Harvard University.” But language is rarely so clear and named entities can be recognized in strings like “the university” or “the Crimson.” An effective NED pipeline would also appropriately link these named entities to the node for “Harvard University.”

Let’s continue trying to match “the university” with the correct knowledge base node. After recognizing that it is a named entity, we can produce a pool of candidate nodes in the knowledge base to which the phrase might be referring. Each candidate node should have a confidence score associated with its estimated probability of being correct. We disambiguate the meaning of “the university” by making a single selection from this candidate pool as our node prediction.

Candidate pools of possible linked nodes are generated for each named entity.

Congruence

A basic version of disambiguation would involve simply selecting the candidate with the highest prior probability independently for each named entity in a unit of text. Our team set out to show that models can perform better disambiguation by incorporating “congruence”, or the contextual relationship between candidates for named entities in the same sentence, and updating our prior scores before making a final prediction. We test methods of calculating congruence that incorporate text similarity, entity similarity and knowledge graph proximity — all different measures of relatedness.

Candidate relatedness should elevate correct answer with low prior probability to become prediction.

Data

AIDA CoNLL-YAGO

Our test dataset is the AIDA CoNLL-YAGO (ACY) manually annotated dataset provided by the Max Planck Institute for Informatics. It was created for the institute’s EMNLP 2011 paper “Robust Disambiguation of Named Entities in Text” and contains recognized named entities and manual Wikipedia page assignments. Using this dataset means that named entity recognition is performed for us and we can focus on the subsequent task of disambiguation. Our goal is to correctly predict the annotated Wikipedia page linked to by each named entity in the ACY dataset.

Kensho-Derived Wikimedia Dataset

In February 2020, Kensho publicly released a refined version of Wikimedia’s open-source knowledge base structured specifically for NLP tasks. The dataset has three distinct layers: the raw English text of a Wikipedia page, hyperlinked Wikipedia text linking to other pages, and a relational knowledge graph of pages and their formal relations to each other. As will be explored later, we use these hyperlinks to understand text-to-page connections and the knowledge graph to explore the relatedness of named entities.

Wikipedia2Vec and Pre-Trained Embeddings

To compare the similarity of named entities, we use the Wikipedia2Vec package developed and maintained by Studio Ousia. This package provides vector embeddings trained over the Wikipedia corpus of data for both words and entities representing entire Wikipedia pages. Vector embeddings are numerical representations of words and/or pages that allow for direct comparisons of relatedness based on their distances in the latent semantic embedding space using for example simple pairwise metrics like cosine or Euclidean distance. Though embeddings can be computationally expensive to fit, transfer learning techniques allow NLP researchers like us to download and deploy highly informative pre-trained embeddings for novel use cases.

Studio Ousia implements a conventional skip-gram model, such as word2vec, to learn the word embeddings. For entity embeddings, three sub-models are jointly optimized as described in Yamada et al. (2016) and summarized in the image below.

Using all three sub-models in conjunction means that the entity embeddings encapsulate both textual context for a named entity as well as graphical relatedness from its knowledge graph. Comparing entity embeddings thus becomes a simpler way of looking at relational similarity while avoiding the high computational complexity that we and other researchers have faced processing large knowledge graphs directly — especially ones as large as Wikipedia.

These embeddings are available in dimension sizes of 100, 300, and 500 and the impact of dimensional depth on performance is explored at various stages in our pipeline.

Pipeline Overview

First, we structure the AIDA CoNLL-YAGO dataset to include information on our defined context window, including a unique sentence_id for each sentence and a list of the named entities within the same sentence.

Data frame enriched with contextual clues for Named Entity Congruence

Second, we generate candidate pools for each named entity that provide the five most probable Wikipedia pages and the associated prior probability of each candidate page. We test two methods of candidate pool generation: A) text<>entity vector similarity and B) anchor link statistics.

Third, we calculate named entity congruence, update the prior probabilities of the isolated candidates with this relatedness metric, and make a disambiguation prediction for each named entity. We test three methods of congruent disambiguation: A) combination centrality, B) entity<>entity vector similarity, and C) knowledge graph proximity.

Overview of Research Pipeline

Candidate Pool Generation

For a named entity in a unit of text, we generate a list of possible candidates for Wikipedia pages (entities) that the named entity might be referring to. We rank these candidates by a calculated prior probability of being the correct choice.

Text<>Entity Vector Similarity

This method identifies candidate links by calculating similarity between the embedding representation of the named entity text string and that of the Wikipedia page entity. We use the Wikipedia2Vec API to perform these similarity calculations through its function most_similar_by_vector.

Example query of Wikipedia2Vec API.

The function requires searching the text string in Wikipedia2Vec’s vocabulary to return the word representation (in its class or vector form) before querying with that. This restriction presents two challenges. First, a word in our test dataset, particularly an uncommon name, may be “out-of-vocabulary” and not appear in Wikipedia2Vec’s trained corpus of words. In this case it will fail to return a result. Second, to handle multi-word named entities, we must take the mean of each component word’s vector representation (like in the above example) and search with this derived vector.

As a result of these two challenges, this method proved unsuccessful in comparing the named entity text vector to the Wikipedia entity vectors. Using 100d embeddings, we produced candidate pools containing the right answer only 34.9% of the time. If we take the most similar candidate in that pool as the prediction, we achieved a predictive accuracy of just 18.82%.

Increasing vector complexity did not help. Both 300 and 500 dimensional embeddings were unable to get us above 50% coverage of the right answer. Coverage in this context means that the candidate pool contains the correct entity, even if it wasn’t our final prediction.

Representing the named entity as a word embedding was insufficiently informative to provide accurate matches to our pre-trained entity embeddings.

Anchor Link Statistics

This second method uses Wikipedia’s inherent data structure to understand how people might refer to a page. An “anchor link” is a text string within a Wikipedia page that hyperlinks to a different Wikipedia page.

Sample anchor link for “Harvard Yard”

The Kensho-derived Wikimedia Dataset structures these anchor links to be easily searchable. We further normalize anchor links to better isolate different ways of referencing the same page and ensuring better comparison with a named entity text string. We calculate, for every anchor link-to-Wikipedia page pair, the frequency of that anchor text linking to the mapped page and the overall viewership popularity of the linked page. These statistics allow us to declare certain pair links as more probable than others by translating the number of times “the university” refers to the Wikipedia page for “Harvard University” into a percentage of the number of times “the university” refers to any page. This fraction is the prior probability that the anchor text “the university” links to the entity “Harvard University.”

Processed anchor links data frame with prior probabilities.

Both definitions of prior probability — the frequency a string links to a page or the popularity of that page’s views — lead to high coverage rates; link frequency achieves 90% coverage and page popularity achieves 86%. Taking the most probable candidate in the pool as our prediction, link frequency and page popularity achieve 72.03% and 61.87% predictive accuracy.

Congruent Prediction

We continue with only the candidate pools generated by anchor link statistics for link frequency. We next apply various congruence methods to improve disambiguation prediction when selecting the best candidate for each named entity. We experimented with both indirectly and directly incorporating prior probabilities.

Correct answer in the top 2 ranks more than 90% of the time.

Indirectly incorporating the prior involves treating candidates equivalently once they’ve passed the threshold to be in the Top N. Directly incorporating priors involves mathematically combining candidate probabilities with a candidate set’s congruence to discount or update the final probability. In all cases, directly incorporating priors outperformed indirect incorporation given the large top rank skew provided by anchor link statistics. As can be seen in the image, the right answer is in the top position 9 times out of 10, with an average prior probability of 88%.

Combination Centrality

Our first method involved the identification of every unique combination of candidates across a sentence’s named entities and the calculation of a representative “centroid vector” of the embeddings in that combination. We then identify the distance between a combination’s every candidate and that combination’s centroid and select the combination with the lowest average distance as our prediction.

This method performed poorly when using cosine similarity as the pairwise distance score, but improved when using Euclidean distance. Despite this improvement, overall predictive accuracy still dropped by two percentage points to 70%. A possible explanation is the vector space is so high-dimensional that cosine angles have infinitesimal variations, or that taking an overall average reduces the impact of highly confident pairs that provide the strongest signal for accurate predictions.

Entity Vector Similarity

Our second method involved iteratively identifying the most similar entities using entity<>entity vector similarity, which incorporates graph proximity in the original fitting of the embedding vectors as described earlier, to calculate the congruence metric. We recursively calculate all pairwise candidate congruence for all named entities within a single sentence, discount the congruence metric by the mean prior probability of that pair, and iteratively predict the next most congruent candidate for each named entity. Specifically, we first identify the most congruent pair and then, assuming that pair to be true, calculate the most congruent candidate to already-predicted candidates until a single best candidate has been chosen for all named entities.

Step One: Identify most congruent pair. Step Two: Identify next most congruent candidate to pair.

This method of congruence using 100d embeddings improves predictive accuracy of named entity disambiguation by a range of 2.4–3.7% depending on the number of named entities within the sentence.

Our candidate pool generation method places the correct answer within the list of possibilities around 90% of the time. This becomes our maximum possible predictive accuracy in the subsequent stage because no matter how good our congruence method is, if the right answer isn’t in the candidate pool, it can never be selected. To achieve perfect accuracy, we would need to improve both coverage and congruent prediction.

Varying number of named entities within a sentence varies predictive accuracy

We see a maximum predictive accuracy with 100-dimensional vector embeddings of 75.795%. Increasing the dimensionality increases performance, but only to a limit. 300d vector embeddings achieved a maximum performance of 76.304%, while 500d vector embeddings achieved 76.291%.

We experimented with a non-iterative approach where we did not restrict the third named entity to predicting off the previously identified most congruent pair, but instead allowed it to potentially pair with the fourth or higher named entity. This reduced top performance by around 2%, suggesting the predictive signal from the highest congruent pair is important to leverage when evaluating over an entire sentence.

Lastly, we experimented with differently sized candidate pools. Providing between 5–10 candidates to the congruence calculation proved the most performant. Less than that lowered maximum performance by 1–2%. More than that (including allowing all candidates to remain) lowered maximum performance by 3% while greatly increasing computation time.

Graph Proximity

Our third method involved locating our candidates within the Kensho-derived knowledge graph and calculating the shortest path distance between all candidates before selecting the closest as our predictions. This method was the most computationally challenging. The Wikipedia knowledge graph has 141 million relations and 50 million nodes and was impossible to load in its entirety into our chosen graph package NetworkX.

To circumvent this, we designed a method we have named “Dendrite Distance.” This approach develops at run-time a much smaller graph by searching for all candidates of named entities in the same sentence, returning the first-degree connections and using this expanded pool as our entire graph. This guarantees to at least provide the shortest path between the closest candidates, since if a candidate pair does not overlap through their first-degree connections, they will not have a shorter path through their second-degree connections. Since our simple approach was only looking for the closest, we determined this to be sufficient.

Step One: Return first degree connections. Step Two: Calculate shortest path within filtered “Dendrite Graph”. Step Three: Select closest pair of candidates.

The involved nature of this approach meant that returning a prediction for a single named entity took between 18–25 seconds. For a small sample of 200 named entities, we saw predictive accuracy of 60.163%. This is lower than using anchor link statistics alone, but we believe there is significant opportunity to improve upon our approach to achieve faster computation, more informational distance metrics, and to directly incorporate our prior confidence by mathematically combining it with the shortest path.

Lastly, we tried to isolate the informational power of the knowledge graph by calculating vector similarity using pre-trained embeddings from the GraphVite package that were developed using just Wikipedia’s knowledge graph compared with Wikipedia2Vec’s three sub-models. Using these knowledge graph-only embeddings for a small sample of 100 produced an accuracy score of 74.528%, an improvement over anchor link statistics alone but not matching the maximum performance exhibited by the more informational embeddings from Wikipedia2Vec.

Summary of research results across methods

Future Research

Our work has shown that there is potential in using Named Entity Congruence, but to expand beyond Wikipedia, a more general method of training informative embeddings like those provided by Wikipedia2Vec must be developed. Research should be done into the performance gains produced by these entity embeddings trained over knowledge bases of various sizes. Since Wikipedia is the world’s largest public knowledge base, will entity embeddings perform similarly well at disambiguation if trained in the same way but over a much smaller private knowledge base?

There is significant room for improvement in the computational performance of graph calculations using specialized technologies like Neo4J. Our “Dendrite Distance” approach can serve as a useful framework upon which to advance the pipeline design, but more focused efforts should be made on optimization.

Lastly, the coverage provided by anchor link statistics could be improved to raise the ceiling above the current ~90% level. Potential paths include better text normalization, incorporating a database of more colloquial hyperlinks than Wikipedia and ensuring multi-language anchor links in the source for uncommon non-English names.

Summary

In this research, we show that incorporating the concept of Named Entity Congruence, or the relatedness between named entities within the same contextual window, can improve predictive accuracy at the NLP task of Named Entity Disambiguation.

Using Wikipedia2Vec’s 300d pre-trained entity embeddings, our congruence method of iterative entity<>entity embedding similarity produced a meaningful absolute gain of 4.2% to achieve overall predictive accuracy of 76.3% on the AIDA CoNLL-YAGO test dataset.

We have proposed a two-part framework of 1) Candidate Pool Generation and 2) Congruent Disambiguation within which future researchers can target specific ideas for improvement. We have provided two different methods to generate candidate pools and three methods to calculate congruence and their associated performance results.

Using Entity<>Entity Vector Similarity as our congruence measure, we have demonstrated that the incorporation of prior probability with congruence can lead to a meaningful increase in predictive accuracy over a well-structured knowledge base like Wikipedia. We hope this starting framework inspires others to include congruence within their NLP pipelines.