Venue: NAACL 2018

# ELDEN: Improved Entity Linking using Densified Knowledge Graphs

 Priya Radhakrishnan\({}^{*}\), Partha Talukdar\({}^{*}\) and Vasudeva Varma\({}^{+}\)

###### Abstract

Entity Linking (EL) systems aim to automatically map mentions of an entity in text to the corresponding entity in a Knowledge Graph (KG). Degree of connectivity of an entity in the KG directly affects an EL system's ability to correctly link mentions in text to the entity in KG. This causes many EL systems to perform well for entities well connected to other entities in KG, bringing into focus the role of KG density in EL. In this paper, we propose Entity Linking using Densified Knowledge Graphs (ELDEN). ELDEN is an EL system which first densifies the KG with co-occurrence statistics from a large text corpus, and then uses the densified KG to train entity embeddings. Entity similarity measured using these trained entity embeddings result in improved EL. ELDEN outperforms state-of-the-art EL system on benchmark datasets. Due to such densification, ELDEN performs well for sparsely connected entities in the KG too. ELDEN's approach is simple, yet effective. We have made ELDEN's code and data publicly available.

1844

## 1 Introduction

Entity Linking (EL) is the task of mapping _mentions_ of an _entity_ in text to the corresponding entity in Knowledge Graph (KG) Hoffwart et al. (2011); Dong et al. (2014); Chisholm and Hachey (2015). EL systems primarily exploit two types of information: (1) similarity of the mention to the candidate entity string, and (2) coherence between the candidate entity and other entities mentioned in the vicinity of the mention in text. Coherence essentially measures how well the candidate entity is connected, either directly or indirectly, with other KG entities mentioned in the vicinity Milne and Witten (2008); Globerson et al. (2016). In the state-of-the-art EL system by Yamada et al. (2016), coherence is measured as distance between embeddings of entities. This system performs well on entities which are densely-connected in KG, but not so well on sparsely-connected entities in the KG.

We demonstrate this problem using the example sentence in Figure 1. This sentence has two mentions: _Andrei Broder_ and _WWW_. The figure also shows mention-entity linkages, i.e., mentions and their candidate entities in KG. Using a conventional EL system, the first mention _Andrei Broder_1 can be easily linked to Andrei_Broder using string similarity between the mention and candidate entity strings. String similarity works well in this case as this mention is unambiguous in the given setting. However, the second mention

_WWW_ has two candidates, World_Wide_Web and WWW_conference, and hence is ambiguous. In such cases, coherence measure between the candidate entity and other unambiguously linked entity(ies) is used for disambiguation.

State-of-the-art EL systems measure coherence as similarity between embeddings of entities. The entity embeddings are trained based on the number of common edges in KG2. In our example, common edges are edges World_Wide_Web shares with Andrei_Broder and WGSWW_conference shares with Andrei_Broder. But WWW_conference has less number of edges (it is a sparsely-connected entity) compared to World_Wide_Web. This leads to poor performance3 whereby _WWW_ is erroneously linked to World_Wide_Web instead of linking to WWW_conference.

Footnote 2: Wikipedia Link based Measure (WLM) [15] used in Yamada et al.’s system is based on number of common edges in KG.

Footnote 3: This paper focuses on mention disambiguation. We assume mention and candidate entities are detected already.

In this paper, we propose ELDEN, an EL system which increases nodes and edges of the KG by using information available on the web about entities and _pseudo entities_. Pseudo Entities are words and phrases that frequently occur in Wikipedia, and co-occur with mentions of KG entities in the web corpus. Thus ELDEN uses a web corpus to find pseudo entities and refines the co-occurrences with Pointwise Mutual Information (PMI) [40] measure. EL-DEN then adds edges to the entity from pseudo entities. In Figure 1, pseudo entity _Program Committee_ co-occurs with mentions of Andrei_Broder and WWW_conference in web corpus and has a positive PMI value with both. So EL-DEN adds edges from _Program Committee_ to Andrei_Broder and WWW_conference, densifying neighborhood of the entities. Coherence, now measured as similarity between entity embeddings where embeddings are trained on densified KG, leads to improved EL performance.

Density (number of KG edges) of candidate entity affects EL performance. In our analysis of density and number of entities having that density in the Wikipedia KG, we find that entities with 500 edges or less make up more than 90%. Thus, creating an EL system that performs well on densely as well as sparsely-connected entities is a challenging, yet unavoidable problem.

We make the following contributions:

* ELDEN presents a simple yet effective graph densification method which may be applied to improve EL involving any KG.
* By using pseudo entities and unambiguous mentions of entity in a corpus, we demonstrate how non-entity-linked corpus can be used to improve EL performance.
* We have made ELDEN's code and data publicly available4. Footnote 4: [https://github.com/priyaradhakrishnan0/ELDEN](https://github.com/priyaradhakrishnan0/ELDEN)

## 2 Related Work

**Entity linking**: Most EL systems use coherence among entities [67] to link mentions. We studied coherence measures and datasets used in six recent5 EL systems [8, 14, 15, 16, 17, 18, 19]. We see that the two popular datasets used for evaluating EL [16] on documents are CoNLL [15] and TAC2010 [11], here after TAC. The popular coherence measures used are (1) WLM, (2) Entity Embedding Similarity and (3) Jaccard Similarity [14, 15]. WLM is widely acknowledged as most popular [11] with almost all the six approaches analyzed above using WLM or its variants. Entity embedding similarity [15] is reported to give highest EL6 performance and is the baseline of ELDEN. Footnote 5: (Shen et al., 2015) presents a survey of EL systems.

**Enhancing entity disambiguation**: Among methods proposed in literature to enhance entity disambiguation utilizing KG [16] uses semantic relatedness between entities in other KGs; and [14] uses paths consisting of defined relations between entities in the KG (IMDB and DBLP). All these methods utilize structured information, while our method shows how unstructured data (web corpus about the entity to be linked) can be effectively used for entity disambiguation.

**Entity Embeddings**: ELDEN presents a method to enhance embedding of entities and words in a 

common vector space. Word embedding methods like word2vec Mikolov et al. (2013) and Glove Pennington et al. (2014) have been extended to entities in EL Yamada et al. (2016); Fang et al. (2016); Zwicklbauer et al. (2016); Huang et al. (2015). These methods use data about entity-entity co-occurrences to improve the entity embeddings. In ELDEN, we improve it with web corpus co-occurrence statistics. Ganea and Hofmann (2017) present a very interesting neural model for jointly learning entity embedding along with mentions and contexts.

**KG densification with pseudo entities**: KG densification using external corpus has been studied by Kotnis _et al._ (2015) and Hegde and Talukdar (2015). Densifying edge graph is also studied as 'link prediction' in literature Martinez et al. (2016). Kotnis _et al._ augment paths between KG nodes using 'bridging entities' which are noun phrases mined from an external corpus. ELDEN has a similar approach as it proposes densifying the KG edges of entities by adding edges from pseudo entities. However, densification is used for relation inference in the former methods whereas it is used for entity coherence measurement in ELDEN.

**Word co-occurrence measures**: Chaudhari _et al._ (2011) survey several co-occurrence measures for word association including PMI, Jaccard Dice (1945) and Co-occurrence Significance Ratio CSRDamani (2013) proves that considering corpus level significant co-occurrences, PMI is better than others. Budiu _et al._ (2007) compare Latent Semantic Analysis (LSA), PMI and Generalized Latent Semantic Analysis (GLSA) and conclude that for large corpora like web corpus, PMI works best on word similarity tests. Hence, we chose PMI to refine co-occurring mentions of entities in web corpus.

## 3 Definitions and Problem Formulation

In this section, we present a few definitions and formulate the EL problem.

**Knowledge Graph** (KG): A Knowledge Graph is defined as \(G=(E,F)\) with entities \(E\) as nodes and \(F\) as edges. In allegiance to EL literature and baselines Milne and Witten (2008); Globerson et al. (2016), we use the Wikipedia hyperlink graph as the KG in this paper, where nodes correspond to Wikipedia articles and edges are incoming links from one Wikipedia article to another. ELDEN ultimately uses a densified version of this Wikipedia KG, as described in Section 4.

**Sparsely connected entities**: Following Hoffart _et al._ (2012), we define an entity to be a sparsely connected entity, if the number of edges incident on the entity node in the KG is less than threshold \(\eta^{7}\). Otherwise, the entity is called a densely-connected entity.

**Entity Linking (EL)**: _Given a set of mentions \(M_{D}=\left\{m_{1},...,m_{n}\right\}\) in a document \(D\), and a knowledge graph \(G=(E,F)\), the problem of entity linking is to find the assignment \(\Lambda:M_{D}\to E_{D}\), where \(E_{D}\) is the set of entities linked to mentions in document \(D\) such that \(E_{D}\subseteq E\)._

For mention \(m_{i}\in M_{D}\), let the set of possible entities it can link to (candidate entities) be \(\mathbb{C}_{i}\). Then, the solution to the EL problem is an assignment \(\Lambda\) where,

\[\Lambda(m_{i})=\operatorname*{arg\,max}_{e\in\mathbb{C}_{i}}\left[\phi(m_{i}, e)+\beta\cdot\psi(e,E_{D})\right] \tag{1}\]

Here, \(\phi(m_{i},e)\in[0,1]\) measures the contextual compatibility of mention \(m_{i}\) and entity \(e\). \(\phi(m_{i},e)\) is obtained by combining prior probability and context similarity. \(\psi(e,E_{D})\) measures the coherence of \(e\) with entities in \(E_{D}\). \(\beta\) is a variable controlling inclusion of \(\psi\) in the assignment \(\Lambda\).

**Problem Formulation :**Yamada et al. (2016) is a recently proposed state-of-the-art EL system. We consider it as a representative EL system and use it as the main baseline for the experiments in this paper. In this section, we briefly describe Yamada _et al._ (2016)'s two-step approach that solves the EL problem presented above.

_Step 1:_ A mention \(m_{i}\in M_{D}\) is defined to be _unambiguous_ if \(\exists e\in\mathbb{C}_{i}\) such that \(\phi(m_{i},e)\geq\gamma\). Let \(M_{D}^{(u)}\subseteq M_{D}\) be the set of such unambiguous mentions in document \(D\). For all unambiguous mentions \(m\in M_{D}^{(u)}\), Yamada _et al._ freeze the assignment by solving Equation 1 after setting \(\beta=0\). In other words, \(\psi(e,E_{D})\) is not used while assigning entities to unambiguous mentions. Assigning entities first to unambiguous mentions has also been found to be helpful in prior research Milne and Witten (2008); Guo and Barbosa (2014). Let \(A_{D}\) be the set of entities linked to in this step. In Figure 1, mention _Andrei Broder_ is unambiguous8.

 

_Step 2:_ In this step, Yamada _et al._ links all ambiguous mentions by solving Equation 2.

\[\Lambda(m_{i})=\\ \arg\max_{e\in\mathbb{C}_{i}}\left(\phi(m_{i},e)+\frac{1}{|A_{D}|} \sum_{e_{j}\in A_{D}}v_{e}\cdot v_{e_{j}}\right)\\ \forall m_{i}\in M_{D}\setminus M^{(u)}_{D} \tag{2}\]

where \(v_{e},v_{e_{j}}\in\mathbb{R}^{d}\) are \(d\)-dimensional embeddings of entities \(e\) and \(e_{j}\) respectively. Please note that the equation above is a reformulation of Equation 1 with \(\beta=1\) and \(\psi(e,E_{D})=\frac{1}{|A_{D}|}\sum_{e_{j}\in A_{D}}v_{e}\cdot v_{e_{j}}\), where \(A_{D}\) is derived from \(E_{D}\) as described in Step 1. As coherence \(\psi(e,E_{D})\) is applied only in disambiguation of ambiguous mentions, we apply densification to only selective nodes of KG.

Embeddings of entities are generated using word2vec model and trained using WLM (Details in Section 5). Given a graph \(G=(E,F)\), the WLM coherence measure \(\psi_{wlm}(e_{i},e_{j})\) between two entities \(e_{i}\) and \(e_{j}\) is defined in Equation 3 where \(C_{e}\) is the set of entities with edge to entity \(e\).

\[\psi_{\mathrm{wlm}}(e_{i},e_{j})=1-\\ \frac{log(max(|C_{e_{i}}|,|C_{e_{j}}|))-log(|C_{e_{i}}\cap C_{e_{ j}})|)}{log(|E|)-log(min(|C_{e_{i}}|,|C_{e_{j}}|))} \tag{3}\]

## 4 Our Approach: ELDEN

In this section, we present ELDEN, our proposed approach. ELDEN extends (Yamada et al., 2016), with one important difference: rather than working with the input KG directly, ELDEN works with the densified KG, created by selective densification of the KG with co-occurrence statistics extracted from a large corpus. Even though this is a simple change, this results in improved EL performance. The method performs well even for sparsely connected entities.

**Overview :** Overview of the ELDEN system is shown in Figure 2. ELDEN starts off with densification of the input KG, using statistics from web corpus. Embeddings of entities are then learned utilizing the densified KG in the next step. Embedding similarity estimated using the learned entity embeddings is used in calculating coherence measure in subsequent EL. Notation used is summarized in Table 1.

_(i)KG Densification_ Figure 2 depicts densification of KG in 'Input KG' and 'Desnified KG'. It shows two Wikipedia titles Andrei_Broder and WWW_conference from our running example (Figure 1). There are no edges common between Andrei_Broder and WWW_conference. In a web corpus, mentions of Andrei_Broder and WWW_conference co-occur with _Program committee_ and it has a positive PMI value with both the entities. So ELDEN adds an edge from _Program committee_ to both the entities. Here _Program committee_ is a pseudo entity. Thus, ELDEN densifies the KG by adding edges from pseudo entities when the mentions of Wikipedia entity and pseudo entity co-occur in a web corpus and the pseudo entity has a positive PMI value with given entity.

Taking a closer look, KG densification process starts from 'input KG' which is Wikipedia hyperlink graph \(G=(E,F)\), where the nodes are Wikipedia titles (\(E\)) and edges are hyperlinks (\(F\)). ELDEN processes Wikipedia text corpus and identifies phrases (unigrams and bi-grams) that occur frequently, i.e. more than 10 times in it. We denote these phrases as _pseudo entities_ (\(S\)) and add them as nodes to the KG. Let \(E_{+}=E\cup S\) be the resulting set of nodes.

ELDEN then adds edges connecting entities in \(E_{+}\) to entities in \(E\). This is done by processing a web text corpus looking for mentions of entities in \(E_{+}\), and linking the mentions to entities in KG \(G^{{}^{\prime}}=(E_{+},F)\). ELDEN uses Equation 1 with \(\beta=0\) for this entity linking, i.e. only mentionity similarity \(\phi(m,e)\) is used during this linking9. Based on this entity linked corpus, a co-occurrence matrix \(M\) of size \(|E_{+}|\times|E_{+}|\) is constructed. Each cell \(M_{i,j}\) is set to the PMI between

\begin{table}
\begin{tabular}{l|p{87.4pt}} \hline
**Notation** & **Definition** \\ \hline \(v_{c}\) & entity embedding of entity \(e\) with dimension \(1*d\) \\ \(E\) & set of all titles in Wikipedia \\ \(S\) & set of all pseudo entities considered in ELDEN \\ \(E_{+}\) & set of all entities considered in ELDEN \\ \(\mathbf{V}\) & word vectors of \(E_{+}\) where \(\mathbf{V}\in\mathbb{R}^{k*d}\) \\ \(\psi\)ELDEN & Embedding similarity measured using \(\mathbf{V}\) trained on \(G_{\text{dense}}\) \\ \(\psi_{\text{Yamada}}\) & Embedding similarity measured using \(\mathbf{V}\) trained on input KG \(G\) \\ \hline \end{tabular}
\end{table}
Table 1: Notation used in KG densification and learning entity embeddings (Please see Sec 4 for more details).

 

the entities \(e\) and \({e^{{}^{\prime}}}\).

\[M_{e,e^{{}^{\prime}}}=\mathrm{PMI}(e,{e^{{}^{\prime}}})=\log\frac{f( e,e^{{}^{\prime}})\times N}{f(e)\times f(e^{{}^{\prime}})}\]

where \(f(e)\) is the frequency of entity \(e\) in web corpus, \(f(e,e^{{}^{\prime}})\) is the sentence-constrained pair frequency of the entity pair \((e,e^{{}^{\prime}})\) in web corpus, and \(N=\sum_{e,e^{{}^{\prime}}\in E_{+}}f(e,e^{{}^{\prime}})\). Please note that PMI, and there by \(M\), are symmetric. The expanded set of edges, \(F_{+}\), is now defined as

\[F_{+}=F\cup\{(e,e^{{}^{\prime}}),(e^{{}^{\prime}},e)\,|\,e^{{}^{\prime}}\in E _{+},e\in E,M_{e,e^{{}^{\prime}}}>0\}\]

In other words, we augment the set of initial edges \(F\) with additional edges connecting entities in \(E_{+}\) with entities in \(E\) such that PMI between the entities is positive.

ELDEN now constructs the KG \(G_{\rm dense}=(E_{+},F_{+})\), which is a densified version of the input KG \(G=(E,F)\). ELDEN uses this densified KG \(G_{\rm dense}\) for subsequent processing and entity linking.

_(ii)Learning Embeddings of Densified KG Entities_ELDEN derives entity embeddings using the same setup, corpus and Word2vec skip-gram with negative sampling model as in Yamada _et al._, However, instead of training embeddings over the input KG, EDLEN trains embeddings of entities in the densified KG \(G_{\rm dense}\). Let **V** be the word2vec matrix containing embeddings of entities in \(E_{+}\) where \(\textbf{V}\in\mathbb{R}^{k*d}\). \(v_{e}\) is the embedding of entity \(e\) in \(E_{+}\) with dimension \(1*d\).

In word2vec model, entities in context are used

Figure 2: ELDEN consists of KG densification, training entity embeddings on densified KG and building EL system that uses similarity between the trained embeddings as coherence measure. ELDEN’s difference from baseline method is that while baseline method uses input KG for training \(v_{e}\), ELDEN uses densified KG for training \(v_{e}\). Hence, the improved performance of ELDEN is solely attributed to densification.

 

to predict the target entity. ELDEN maximizes the objective function (Goldberg and Levy, 2014) of word2vec skip-gram model with negative sampling, \(L=\sum_{(t,c)\in\mathbb{P}}L_{t,c}\) where

\[L_{t,c}=log~{}\theta(v_{c}\cdot v_{t})+\sum_{n\in\mathbb{N}_{(t,c)}}log~{} \theta(-v_{n}\cdot v_{t})\]

Here \(v_{t}\) and \(v_{c}\) are the entity embeddings of target entity \(t\) and context entity \(c\). \(\mathbb{P}\) is the set of target-context entity pairs considered by the model. \(\mathbb{N}_{(t,c)}\) is a set of randomly sampled entities used as negative samples with pair \((t,c)\). This objective is maximized with respect to variables \(v_{t}\)'s and \(v_{c}\)'s, where \(\theta(x)=\frac{1}{1+e^{-x}}\).

\(\mathbb{P}\) and \(\mathbb{N}\) are derived using \(G_{\mathrm{dense}}\). \(t\) and \(c\) are entities in \(E_{+}\) such that \(c\) shares a common edge with \(t\). \(v_{n}\) is randomly sampled from \(\mathbf{V}\), for entities that do not share a common edge with \(t\). Entity embedding similarity measured using \(\mathbf{V}\) trained this way on \(G_{\mathrm{dense}}\) is \(\psi_{\mathrm{ELDEN}}\). Embedding similarity is measured as cosine distance between \(v_{e}\)s. Embeddings of \(S\) are trained using positive and negative word contexts derived using context length.

**(iii) Bringing it All Together: EDLEN**

ELDEN is a supervised EL system which uses two sets of features: (1) contextual compatibility \(\phi(m,e)\); and (2) coherence \(\psi(e_{i},e_{j})\). These features are summarized in Table 2. Similarity between entity embeddings is measured as cosine similarity between \(v_{e}\)s.

## 5 Experiments

In this section, we evaluate the following:

* Is ELDEN's corpus co-occurrence statistics-based densification helpful in disambiguating entities better? (Sec. 6.1)
* Where does ELDEN's selective densification of KG nodes link entities better? (Sec. 6.3)

**Setup :** ELDEN is implemented using Random Forest ensemble 10 (Breiman, 1998). Parameter values were set using CoNLL development set. Feature limit of 3 with number of estimators as 100 yielded best performance.

Footnote 10: [http://scikit-learn.org](http://scikit-learn.org)

**Knowledge Graph: Wikipedia** Following prior EL literature, we use Wikipedia hypergraph as our KG (Milne and Witten, 2008; Globerson et al., 2016). This KG is enhanced with pseudo entities as explained in Section 4. We process the Wikipedia corpus following the same procedure as in Yamada _et al._ (2016). We cleaned 11 Wikipedia dump dated Nov 2015. We then parsed the Wikipedia article text to identify pseudo entities. More details on KG and parameters used for training embeddings are in Table 3. Training took 4 days on gpu with 2 cores.

Footnote 11: by removing disambiguation, navigation, maintenance and discussion pages.

**Preprocessing: Web corpus and Densified KG** For our experiments, we created a web corpus by querying Google12. Candidate entities of all mentions in the dataset are queried in Google and top ten search results are considered for unigram and bigram frequencies. This corpus occupied 6.8GB for candidate entities of TAC and CoNLL dataset mentions (54336 entities). Even for sparsely connected entities, an average corpus size of 670 lines or more13 was collected. We note that though some of the entities mentioned in this dataset are ten or more years old, we are able to collect, on an average more than 670 lines of web content. Thus corpus proves to be a good source of additional links for densification, for both common and rare entities. As Taneva and Weikum (2013) also note, it is not hard to find content about sparsely connected entities on the web.

Footnote 12: [https://www.google.com/](https://www.google.com/)

Footnote 13: A detailed analysis of knowledge gained from crawling for common versus less common entities is present in Figure 1 of supplementary material.

The web corpus is analyzed for mentions and pseudo entities. Co-occurrence matrix \(M\) is created14 for mention and pseudo entities occurring within window of size 10 for PMI calculation15. Edges are added from pseudo entities with positive PMI to mention of given entity. In experiments we add edges from top 10 pseudo entities ordered by

\begin{table}
\begin{tabular}{l|l} \hline
**Specifics** & **Value** \\ \hline Number of titles after cleaning (\(|E|\)) & 4.6 M \\ Number of pseudo entities (\(|S|\)) & 1.3 M \\ Total number of entities (\(|E_{+}|\)) & 5.9 M \\ Number of epochs & 3 \\ Learning rate & 0.25 linearly reducing to 0.01 \\ Number of negative samples (\(|\mathbb{N}|\)) & 5 \\ Context window size & 3 \\ Dimensions of embedding (\(d\)) & 100 \\ \hline \end{tabular}
\end{table}
Table 3: Parameters used in Wikipedia processing and training KG.

 

PMI values16.

Footnote 15: This is a tunable parameter.

**Evaluation Dataset:** In line with prior work on EL, we test the performance of ELDEN on CoNLL and TAC datasets. As this paper focuses on entity disambiguation, we tested ELDEN against datasets and baseline methods for disambiguation. We note that the entity disambiguation evaluation part of other recent datasets like ERD 2014 and TAC 2015 is exactly same as the TAC 2010 evaluation Ellis et al. (2014) 17.

Footnote 16: This is a tunable parameter.

Footnote 17: These recent datasets consist of other evaluations, e.g., mention detection, multilinguality etc. which is beyond the scope of the paper and hence we didn focus on them in the paper.

**Training:** ELDEN's parameters were tuned using training (development) sets of CoNLL and TAC datasets. CoNLL and TAC datasets consist of documents where mentions are marked and entity to which the mention links to, is specified. We use only mentions that link to a valid Wikipedia title (non NIL entities) and report performance on test set. Some aspects of these datasets relevant to our experiments are provided below.

**CoNLL**: In CoNLL test set (5267 mentions), we report Precision of topmost candidate entity, aggregated over all mentions (P-micro) and aggregated over all documents (P-macro), i.e., if tp, fp and p are the individual true positives, false positives and precision for each document in a dataset of \(\delta\) documents, then

\[P_{\mathrm{micro}}=\frac{\sum\limits^{\delta}_{i=1}\mathrm{tp}_{i}}{\frac{ \sum\limits_{i=1}^{\delta}\mathrm{tp}_{i}+\frac{\delta}{\epsilon}\mathrm{fp} _{i}}}\text{ and }P_{\mathrm{macro}}=\frac{\sum\limits^{\delta}_{i=1}\mathrm{p}_{i}}{\delta}.\]

For CoNLL candidate entities, we use Pershina et al. (2015) dataset18.

Footnote 18: [https://github.com/masha-p/PPRforNED](https://github.com/masha-p/PPRforNED)

**TAC**: In TAC dataset, we report P-micro of topranked candidate entity on 1,020 mentions. P-macro is not applicable to TAC as most documents have only one mention as query mention ( or 'mention to be linked'). For TAC candidate entities, we index the Wikipedia word tokens and titles using sdlr19. We index terms in (1) title of the entity, (2) title of another entity redirecting to the entity, and (3) names of anchors that point to the entity, in line with baselines. We are making this TAC candidate set publicly available.

Footnote 19: [http://lucene.apache.org/solt/](http://lucene.apache.org/solt/)

**Baseline: Yamada16** Our baseline is the Yamada system explained in Section 3. Entity embedding distance measured using \(v_{e}\) trained on the input KG \(G\) is \(\psi_{\mathrm{Yamada}}\).

## 6 Results

### Does ELDEN's selective densification help in disambiguation in EL?

In Table 4, we compare ELDEN's EL performance with results of other recently proposed state-of-the-art EL methods that use coherence models. We see that ELDEN results matches best results on CoNLL and outperforms state-of-the-art in TAC dataset. In the table, the last four rows uses the Pershina _et al._Pershina (2015) candidate set and hence, we provide a comparison of their disambiguation performance. Improved results of ELDEN over base-line is attributed to the improved disambiguation due to KG densification.

### Why does ELDEN's selective densification work?

We conduct ablation analysis using various feature and feature combinations and present performance of ELDEN and baseline in Table 5. Starting with base features, we add various features to ELDEN incrementally and report their impact on performance. The results when using base feature group alone, and base and string similarity groups together (\(\phi\)) are presented in first and second rows for each dataset. We compare \(\psi_{\mathrm{ELDEN}}\) to three coherence measures: \(\psi_{\mathrm{wlm}}\), \(\psi_{\mathrm{Yamada}}\) and \(\psi_{\mathrm{dense}}\), details of which are provided in Table 2. The performance improvement from each of the four coherence measures are in the next four rows. Performance of ELDEN from using all four coherence features is given in \(\psi_{\mathrm{ELDEN++}}\) row.

On CoNLL dataset, \(\psi_{\mathrm{dense}}\) combined with \(\phi\)

\begin{table}
\begin{tabular}{l|c c c} \hline \hline
* \\ Chisholm and Hachey (2015) & 88.7 & **-** & 80.7 \\ \hline Pershina et al. (2015) & 91.8 & 89.9 & **-** \\ Globerson et al. (2016) & 91.7 & **-** & 87.2 \\ Yamada et al. (2016) & **93.1** & 92.6 & 85.2 \\
**ELDEN** & 93.0 & **93.7** & **89.6** \\ \hline \hline \end{tabular}
\end{table}
Table 4: Performance comparison with other recent EL approaches. ELDEN matches best results in CoNLL and outperforms the state-of-the-art in TAC dataset. (Please see Section 6.1 for details and \(\psi_{\mathrm{ELDEN++}}\) row of Table 5 for ELDEN results.) 

gave an improvement of 2.0 and 1.9 (P-micro and P-macro) over Yamada16 results. We note that Yamada16 results are from our re-implementation of Yamada et al. (2016) system 20 and we are able to almost reproduce the baseline results. We also present the results combining baseline's \(\psi_{\rm Yamada}\) and \(\psi_{\rm wlm}\) versus ELDEN's \(\psi_{\rm ELDEN}\) and \(\psi_{\rm dense}\) in next two rows. We find the ELDEN's KG densification features perform better than baselines.

Footnote 20: We have re-implemented the Yamada _et al_ system using hyper-parameters specified in the paper and these are our best-effort results.

On TAC dataset also, combined with \(\phi\), \(\psi_{\rm dense}\) is found to do better than \(\psi_{\rm wlm}\) and \(\psi_{\rm ELDEN}\) gives a significant P-micro improvement of 4.2 over \(\psi_{\rm Yamada}\). The \(\psi_{\rm ELDEN++}\) P-micro in TAC dataset is statistically significant21. In short, we find the KG densification features, \(\psi_{\rm dense}\) and \(\psi_{\rm ELDEN}\), as the features causing better performance of ELDEN on both datasets.

Footnote 21: We performed two tailed t-test, with 2-tail 95% value of 1.96.

### Where does ELDEN's selective densification work better?

While most EL systems give higher precision on CoNLL dataset than TAC dataset, ELDEN performs with high precision on TAC dataset too. This is explained by analyzing distribution of densely-connected and sparsely connected entities in TAC and CoNLL datasets as presented in Table 6. We see that CoNLL test set has almost half as densely-connected and half as sparsely connected entities, whereas in TAC test set, 63.6% are sparsely connected entities. This higher constitution of sparsely connected entities in TAC, explains ELDEN's better results in TAC relative to CoNLL dataset. As the number of sparsely connected entities is more than the number of densely-connected entities in most KGs Reinanda et al. (2016), our method is expected to be of significance for most KGs.

### What type of EL errors are best fixed with ELDEN's selective densification ?

We analyzed errors fixed by ELDEN on TAC dataset. We categorize the errors into four classes in line with error classes of Ling _et al._ (2015). We manually analyzed 240 wrong predictions of Yamada16 and compared it with that of ELDEN, and the results are presented in Figure 3. We found errors to reduce with use of KG densification features and most of the errors eliminated were in "Specific label" class. Errors in this class called for better modeling of mention's context and link-based similarity Ling et al. (2015).(More details of this analysis in the supplementary document.)

Figure 3: False positives of ELDEN before and after KG densification. Errors reduce with use of \(\psi_{\rm dense}\) and \(\psi_{\rm ELDEN}\) measures. (Please see Sec 6.4)

\begin{table}
\begin{tabular}{l|c|p{284.5pt}} \hline  & \multicolumn{2}{c}{**\% sparsely connected Entities**} \\ \cline{2-3}
**Dataset** & **Train** & **Test** \\ \hline TAC & 78.8 & 63.6 \\ ConvLL & 48.4 & 48.2 \\ \hline \end{tabular}
\end{table}
Table 6: Percentage of sparsely connected entities in evaluation datasets. TAC has higher composition of sparsely connected entities than CoNLL. Hence, ELDEN results are better in TAC over CoNLL (Please see Table 4).

 

Conclusion

We started this study by analyzing the performance of a state-of-the-art Entity Linking (EL) system and found that its performance was low when linking entities sparsely-connected in the KG. We saw that this can be addressed by densifying the KG with respect to the given entity. We proposed ELDEN, which densities edge graph of entities using pseudo entities and mentions of entities in a large web corpus. Through our experiments, we find that ELDEN outperforms state-of-the-art baseline on benchmark datasets.

We believe that ELDENs combination of KG densification and entity embeddings is novel. Poor performance of EL systems on sparsely connected entities has been recognized as one of the open challenges by prior research. ELDEN performs well on sparsely connected entities too, as a validation of our method of combining KG densification followed by embedding. Our approach may be applied to any KG as the densification is performed with the help of _unstructured data_, and not any specific KG. We hope the simple graph densification method utilized in ELDEN will be of much interest to the research community.

Pseudo entities can be looked at as entity candidates for KG expansion, as also noted by Farid _et al._ (2016). In future, we plan to enhance EL-DEN using EL of pseudo entities to estimate _entity prior_ of entities not present in KG. We also plan to explore entity embeddings obtained using other graph densifying methods.

## 8 Acknowledgments

We thank the Microsoft Research India Travel Grants for generous travel funds to attend and present this paper at NAACL.

## References

* Barrena et al. (2016) Ander Barrena, Aitor Soroa, and Eneko Agirre. 2016. Alleviating poor context with background knowledge for named entity disambiguation. In _ACL (1)_. The Association for Computer Linguistics.
* Bhattacharya and Getoor ???? Online collective entity resolution.
* Breiman (1998) Leo Breiman. 1998. Arcing classifier (with discussion and a rejoinder by the author). _Ann. Statist_. 26(3):801-849. [https://doi.org/10.1214/aos/1024691079](https://doi.org/10.1214/aos/1024691079).
* Budiu et al. (2007) Raluca Budiu, Christiaan Royer, and Peter Pirolli. 2007. Modeling information scent: A comparison of Isa, pmi and glsa similarity measures on common tests and corpora. In _Large Scale Semantic Access to Content Text, Image, Video, and Sound_. Paris, France, France, RIAO '07, pages 314-332. [http://dl.acm.org/citation.cfm?id=1931390.1931422](http://dl.acm.org/citation.cfm?id=1931390.1931422).
* Chaudhari et al. (2011) Dipak Chaudhari, Om P. Damani, and Srivatsan Laxman. 2011. Lexical co-occurrence, statistical significance, and word association. _CoRR_ abs/1008.5287.
* Cheng and Roth (2013) Xiao Cheng and Dan Roth. 2013. Relational inference for wikification. In _EMNLP_. ACL, pages 1787-1796.
* Chisholm and Hachey (2015) Andrew Chisholm and Ben Hachey. 2015. Entity disambiguation with web links. _Transactions of the Association for Computational Linguistics_ 3:145-156. [https://tacl2013.cs.columbia.edu/ojs/index.php/tacl/article/view/494](https://tacl2013.cs.columbia.edu/ojs/index.php/tacl/article/view/494).
* Church and Hanks (1989) Kenneth Ward Church and Patrick Hanks. 1989. Word association norms, mutual information, and lexicography. ACL '89, pages 76-83. [https://doi.org/10.3llis/981623.981633](https://doi.org/10.3llis/981623.981633).
* Damani (2013) Om P. Damani. 2013. Improving pointwise mutual information (PMI) by incorporating significant co-occurrence. _CoRR_ abs/1307.0596.
* Dice (1945) Lee R. Dice. 1945. Measures of the amount of ecologic association between species. _Ecology_ 26(3):297-302. [https://doi.org/10.2307/1932409](https://doi.org/10.2307/1932409).
* Dong et al. (2014) Xin Dong, Evgeniy Gabrilovich, Jeremy Heitz, Wilko Horn, Ni Lao, Kevin Murphy, Thomas Strohmann, Shaohua Sun, and Wei Zhang. 2014. Knowledge vault: A web-scale approach to probabilistic knowledge fusion. ACM, New York, NY, USA, KDD '14, pages 601-610. [https://doi.org/10.1145/2623330.2623623](https://doi.org/10.1145/2623330.2623623).
* Ellis et al. (2014) Joe Ellis, Jeremy Getman, and Stephanie Strassel. 2014. Overview of linguistic resource for the tac kbp 2014 evaluations: Planning, execution, and results. Linguistic Data Consortium, University of Pennsylvania.
* Fang et al. (2016) Weiyi Fang, Jianwen Zhang, Dilin Wang, Zheng Chen, and Ming Li. 2016. Entity disambiguation by knowledge and text jointly embedding. In _CoNLL_.
* Farid et al. (2016) Mina Farid, Ihab F. Ilyas, Steven Euijong Whang, and Cong Yu. 2016. Lonlies: Estimating property values for long tail entities. In _Proceedings of the 39th International ACM SIGIR Conference on Research and Development in Information Retrieval_. ACM, New York, NY, USA, SIGR '16, pages 1125-1128. [https://doi.org/10.1145/2911451.2911466](https://doi.org/10.1145/2911451.2911466).
* Ganea and Hofmann (2017) Octavian-Eugen Ganea and Thomas Hofmann. 2017. Deep joint entity disambiguation with local neural attention. In _Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing_. Association for Computational Linguistics, pages 2609-2619. [http://www.aclweb.org/anthology/D17-1276](http://www.aclweb.org/anthology/D17-1276).
* Globerson et al. (2016) Amir Globerson, Nevena Lazic, Soumen Chakrabarti, Amarag Subramanya, Michael Ringaard, and Fernando Pereira. 2016. Collective entity resolution with multifocal attention. In _ACL_.
* Goldberg and Levy (2014) Yoav Goldberg and Omer Levy. 2014. word2vec explained: deriving mikolov et al.'s negative-sampling word-embedding method. _CoRR_ abs/1402.3722. [http://arxiv.org/abs/1402.3722](http://arxiv.org/abs/1402.3722) 

Stephen Guo, Ming-Wei Chang, and Emre Kiciman. 2013. To link or not to link? a study on end-to-end tweet entity linking. In _NAACL-HLT 2013_. [http://research.microsoft.com/apps/pubs/default.aspx?id=183909](http://research.microsoft.com/apps/pubs/default.aspx?id=183909).
* Guo and Barbosa (2014) Zhaochen Guo and Denilson Barbosa. 2014. Robust entity linking via random walks. ACM, New York, NY, USA, CIKM '14, pages 499-508. [https://doi.org/10.1145/2661829.2661887](https://doi.org/10.1145/2661829.2661887).
* Han and Zhao (2010) Xianpei Han and Jun Zhao. 2010. Structural semantic relatedness: A knowledge-based method to named entity disambiguation. In _Proceedings of the 48th Annual Meeting of the Association for Computational Linguistics_. Association for Computational Linguistics, Stroudsburg, PA, USA, ACL '10, pages 50-59. [http://dl.acm.org/citation.cfm?id=1858681.1858687](http://dl.acm.org/citation.cfm?id=1858681.1858687).
* He et al. (2013) Zhengyan He, Shujie Liu, Yang Song, Mu Li, Ming Zhou, and Houfeng Wang. 2013. Efficient collective entity linking with stacking. EMNLP. [http://research.microsoft.com/apps/pubs/default.aspx?id=202249](http://research.microsoft.com/apps/pubs/default.aspx?id=202249).
* Hegde and Talukdar (2015) Manjunath Hegde and Partha P. Talukdar. 2015. An entity-centric approach for overcoming knowledge graph sparsity. Association for Computational Linguistics, Lislbon, Portugal, EMLNLP '15, pages 530-535. [http://aclweb.org/anthology/D15-1061](http://aclweb.org/anthology/D15-1061).
* Hoffart et al. (2012) Johannes Hoffart, Stephan Seufert, Dat Ba Nguyen, Martin Theobald, and Gerhard Weikum. 2012. Kore: Keyphrase overlap relatedness for entity disambiguation. ACM, New York, NY, USA, CIKM '12, pages 545-554. [https://doi.org/10.1145/2396761.2396832](https://doi.org/10.1145/2396761.2396832).
* Hoffman et al. (2011) Johannes Hoffart, Mohamed Amir Yosef, Ilaria Bordino, Bagen Furstenau, Manfred Pinkal, Marc Spaniol, Bilyana Taneva, Stefan Thater, and Gerhard Weikum. 2011. Robust disambiguation of named entities in text. Association for Computational Linguistics, Stroudsburg, PA, USA, EMNLP '11, pages 782-792. [http://dl.acm.org/citation.cfm?id=2145432.2145521](http://dl.acm.org/citation.cfm?id=2145432.2145521).
* Huang et al. (2015) Hongzhao Huang, Larry Heck, and Heng Ji. 2015. Leveraging deep neural networks and knowledge graphs for entity disambiguation. _CoRR_ abs/1504.07678. [http://arxiv.org/abs/1504.07678](http://arxiv.org/abs/1504.07678).
* Ji et al. (2010) Heng Ji, Ralph Grishman, Hoa Tang Dang, Kira Griffitt, and Joe Ellis. 2010. Overview of the tac 2010 knowledge base population track. In _In Third Text Analysis Conference (TAC)_.
* Kotnis et al. (2015) Bhushan Kotnis, Pradeep Bansal, and Partha P. Talukdar. 2015. Knowledge base inference using bridging entities. In _Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, EMNLP 2015, Lisbon, Portugal, September 17-21, 2015_. pages 2038-2043. [http://aclweb.org/anthology/D/D15/D15-1241.pdf](http://aclweb.org/anthology/D/D15/D15-1241.pdf).
* Ling et al. (2015) Xiao Ling, Sameer Singh, and Dan Weld. 2015. Design challenges for entity linking. _Transactions of the Association for Computational Linguistics (TACL)_ 3.
* Martinez et al. (2016) Victor Martinez, Fernando Berzal, and Juan-Carlos Cubero. 2016. A survey of link prediction in complex networks. _ACM Comput. Surv._ 49(4):69:1-69:33. [https://doi.org/10.1145/3012704](https://doi.org/10.1145/3012704).
* Mikolov et al. (2013) Tomas Mikolov, Ilya Sutskever, Kai Chen, Gregory S. Corrado, and Jeffrey Dean. 2013. Distributed representations of words and phrases and their compositionality. In _NIPS 2013_. pages 3111-3119.
* Milne and Witten (2008) David Milne and Ian H. Witten. 2008. Learning to Link with Wikipedia. ACM, CIKM'08, pages 509-518.
* Pennington et al. (2014) Jeffrey Pennington, Richard Socher, and Christopher D. Manning. 2014. Glove: Global vectors for word representation. In _Empirical Methods in Natural Language Processing (EMNLP)_. pages 1532-1543. [http://www.aclweb.org/anthology/D14-1162](http://www.aclweb.org/anthology/D14-1162).
* Pershina et al. (2015) Maria Pershina, Yifan He, and Ralph Grishman. 2015. _Personalized page rank for named entity disambiguation_. Association for Computational Linguistics (ACL), pages 238-243. NAACL HLT 2015.
* Reinanda et al. (2016) Ridho Reinanda, Edgar Meij, and Maarten de Rijke. 2016. Document filtering for long-tail entities. ACM, CIKM'16.
* Shen et al. (2018) W. Shen, J. Han, J. Wang, X. Yuan, and Z. Yang. 2018. Shine+: A general framework for domain-specific entity linking with heterogeneous information networks. _IEEE Transactions on Knowledge and Data Engineering_ 30(2):353-366. [https://doi.org/10.1109/TKDE.2017.2730862](https://doi.org/10.1109/TKDE.2017.2730862).
* Shen et al. (2015) W. Shen, J. Wang, and J. Han. 2015. Entity linking with a knowledge base: Issues, techniques, and solutions. _IEEE Transactions on Knowledge and Data Engineering_ 27(2):443-460. [https://doi.org/10.1109/TKDE.2014.2327028](https://doi.org/10.1109/TKDE.2014.2327028).
* Sun et al. (2015) Yaming Sun, Lei Lin, Duyu Tang, Nan Yang, Zhenzhou Ji, and Xiaolong Wang. 2015. Modeling mention, context and entity with neural networks for entity disambiguation. In Qiang Yang and Michael Wooldridge, editors, _IJCAI_. AAAAI Press, pages 1333-1339. [http://dblp.uni-trier.de/db/conf/ijcai/ijcai2015.html#SunLTYW15](http://dblp.uni-trier.de/db/conf/ijcai/ijcai2015.html#SunLTYW15).
* Taneva and Weikum (2013) Bilyana Taneva and Gerhard Weikum. 2013. Gem-based entity-knowledge maintenance. ACM, New York, NY, USA, CIKM '13, pages 149-158. [https://doi.org/10.1145/2505515.2505715](https://doi.org/10.1145/2505515.2505715).
* Yamada et al. (2016) Ikuya Yamada, Hiroyuki Shindo, Hideaki Takeda, and Yoshiya Natefuji. 2016. Joint learning of the embedding of words and entities for named entity disambiguation. CoNLL 2016, pages 250-259. [http://aclweb.org/anthology/K/K16/K16-1025.pdf](http://aclweb.org/anthology/K/K16/K16-1025.pdf).
* Zwicklbauer et al. (2016) Stefan Zwicklbauer, Christin Seifert, and Michael Granitzer. 2016. Robust and collective entity disambiguation through semantic embeddings. In _Proceedings of the 39th International ACM SIGIR Conference on Research and Development in Information Retrieval_. ACM, New York, NY, USA, SIGIR '16, pages 425-434. [https://doi.org/10.1145/2911451.2911535](https://doi.org/10.1145/2911451.2911535).