International Journal of Computational Linguistics and Applications, vol. 5, no. 2, 2014, pp. 49–66
Received 2/11/2014, accepted 10/12/2014, final 15/12/2014.
ISSN 0976-0962, http://ijcla.bahripublications.com

Exploring Linguistic Features for
Named Entity Disambiguation
S HUANGSHUANG Z HOU , C ANASAI K RUENGKRAI ,NAOAKI O KAZAKI , AND K ENTARO I NUIGraduate School of Information Sciences, JapanA BSTRACT
Resolving named entities is important for a number of natural
language processing applications. However, a named entity has
multiple name variations while different entities could share thesame surface. State-of-the-art systems are based on a global resolution method and mostly adopt link-based features that leveragerelationships of co-occurring entities in the knowledge. We foundthat linguistic features can also significantly affect disambiguation. In this work, we try to explore important linguistic featuresfrom context, which could be the fundamental part of the combination of global resolution method and effective features. Therefore, we study and compare the effects of linguistic features in acomprehensive way. Moreover, we found effective linguistic features according to the experiment results.
K EYWORDS : Named entity disambiguation, entity linking, feature study.

1

I NTRODUCTION

In natural language processing, named entities are important components.
However, due to various ways of writing, named entities have multiple
surfaces in texts, e.g., Big Blue and IBM. Moreover, different entities often share the same surface. For example, “New York” as a place name has

50

S. ZHOU, C. KRUENGKRAI, N. OKAZAKI, AND K. INUI

dozens of different referents in Wikipedia.1 Thus, resolving name mentions to their corresponding entities is necessary. Named entity disambiguation is the task of identifying whether a mention refers to a certain
entity and linking mentions to their corresponding entries in a large-scale
knowledge base. Therefore, NED (Named entity disambiguation) task isalso known as entity linking task.
Ji et al. [1] summarized two main processes: candidate generation andcandidate ranking. Because of the variety of named entities, when given
a mention, NED systems firstly often generate a candidate list contains
as much as possible candidate entities. Then selecting a correct entity
from the ranked candidate list by using a ranking model is the final purpose. Since the selected entity should be coherent with the context inthe source document, disambiguating entities by leveraging the context
information is a fundamental way [2]. Erbs et al. [3] mentioned that features for candidate ranking could be grouped into: linguistic-based (textin source document and text extracted from the KB titles) and link-based(how entities in the same context link in the knowledge).
State-of-the-art systems [4, 2] simultaneously resolve multiple entities (global inference) and mostly adopt link-based features. Those linkbased features measure the relatedness of co-occurring entities in context. They assume that a candidate entity could be linked if it and its
neighbor entities strongly connect in the knowledge base. For example,
in the text of Figure 1, entities in context like San Antonio Spurs andNational Basketball Association strongly support the [Alvin Robertson]
candidate for the mention Robertson because they are linked in the KB.
In Wikipedia, articles titled San Antonio Spurs and National Basketball
Association have in-links from the article titled Alvin Robertson. At thesame time, the article titled Alvin Robertson also have out-links to articles
titled San Antonio Spurs and National Basketball Association.
However, when there is seldom co-occurring entities in context, linguistic information could affect disambiguation significantly [5, 6]. For
example, in the text of Figure 2, words like sophomore, British universities, and U.S. schools suggest [University of St.Andrew] as the correct
entity for St.Andrew. Therefore, we could capture linguistic features, suchas comparing the topic distribution of source documents and the KB texts.
We find that linguistic features can locally measure the coherence between mentions and entities in context. Therefore, we study the effects ofmultiple linguistic features in a comprehensive way in this paper. Espe1

http://en.wikipedia.org/

EXPLORING LINGUISTIC FEATURES FOR NAMED ENTITY ...

51

Knowledge base

Robertson,	
  who	  was	  an	
  All-­‐Star	
  player	
  for	  the	  San	  Antonio	
  Spurs	
  of	
  
the	
  Na1onal	
  Basketball	
  Associa1on,	
  was	
  taken	
  into	  custody	
  Friday.	
  Alvin	  Robertson	  San Antonio Spurs

National Basketball Association

Fig. 1. Example of documents containing mentions for link-based features
KB
Now Waldrop, of Silver Spring, Md., is a [St. Andrews]mention sophomore, one of a growing numberof American students who enroll at top-ranked British universities, which offer the prestige of elite
U.S. schools at a fraction of the cost.

University of St Andrew
St Andrew, Scotland

Fig. 2. Example of documents containing mentions for linguistic features

cially, we compare the effectiveness of each linguistic feature, e.g., document similarity, document topics, entity-level features, and POS features.
Moreover, several linguistic features are used as local methods bystate-of-the-art global inference systems [4, 2, 7–10]. Considering theirinsufficient local methods, in this paper, we examine the contributions of
linguistic features in order to explore more effective local methods for
global inference methods.
2

R ELATED W ORK

Linguistic features showed promising results in previous studies [5, 6,
11, 12], such as document similarity, word overlapping, entity-level word
overlapping, document topics. However, only partial linguistic features
are explored by previous work. Dredze et al. [6] captured features basedon mentions, source documents and KB entries, but features about document topics are not involved. Zhang et al. [11] made big efforts on candidate selection and acronym expansion, but their disambiguation method
only depended on document topics. Therefore, we summarize and refine
effective linguistic features of previous work, and propose a broad rangeof linguistic features in this paper.

52

S. ZHOU, C. KRUENGKRAI, N. OKAZAKI, AND K. INUI

Although some previous work reviewed various ranking methods (unsupervised or supervised) and evaluation results [13, 1], they lack comparing effects of linguistic features systemically. Moreover, Garcia etal. [14] systemically reviewed and evaluated several state-of-the-art linkbased approaches, but they did not mention linguistic-based context features. To our knowledge, previous work did not examine linguistic features comparatively. Therefore, we try to explore context information on
the linguistic level in a comprehensive way.
On the other hand, link-based features strongly depend on the link
structure of knowledge base (Wikipedia), e.g., link statistics (incoming
links and outgoing links), and category information, etc. Link-based features are mostly used by global inference systems for candidate ranking. Relatedness is widely used by [8, 15–17, 2, 4], which is to computethe similarity between two KB entries based on the in/out links. Relatedness is effective to measure the relationship between candidates andco-occurring entities in context.
Linguistic features are used to measure the coherence between mentions and candidates, which are also called local methods by previous
studies [2, 7, 18, 9, 10]. Combining local methods with global features or
global ranking methods, the NED system performance is improved significantly [2]. Among of them, TF/IDF cosine similarity is mostly usedby global inference systems for multiple purposes: ranking candidates [2,7], filtering out noisy candidate [18], and assigning an initial confidence
score for subsequent ranking phrase [9, 10]. However, TF/IDF cosine
similarity is insufficient to capture the coherence between mentions andentities. Moreover, entity popularity is a salient measure of mentions andentities [8, 19–21], and it could check how likely a surface refer to anentity. Entity popularity is a strong baseline for entity linking [14]. However, this feature could ignore unpopular correct entities.
Therefore, our further motivation is to explore useful linguistic features for global inference. Based on our local features, graph-based methods are applied on short and high-coverage candidate lists, at the sametime, unpopular entities can not be missed the candidate list.

3

S YSTEM A RCHITECTURE

The TAC KBP entity linking task provides high-quality data set and comprehensive evaluation. The data set contains a query file, a collection ofsource documents, and a reference KB.

EXPLORING LINGUISTIC FEATURES FOR NAMED ENTITY ...

53

In mention query files, information about one mention is given: thename surface, the document ID, and the position of this mention in thesource document (UTF-8 character offsets). For example,<q u e r y i d =” EDL14 ENG TRAINING 3091”><name>S t . Andrews </name><d o c i d >WPB ENG 20101221 .0 03 1 </ d o c i d ><beg >1123</ beg>
<end >1133</ end></ q u e r y >The example texts in Figure 2 are from source documents. The TAC KBP
official reference KB is extracted from an October 2008 dumps of English
Wikipedia and consists of 818,741 entries.
Systems are required to generate a link-ID file, which contains pairsof a query and the resolved result (corresponding KB entry ID or NIL).
For example, system should output a KB ID e.g., “E0127848” or NIL
for the query “EDL14 ENG TRAINING 3091”. In this task, NIL means
mentions that do not have entries in the KB. TAC KBP added the mention
detection task in 2014. A system should detect possible mentions in raw
documents.
We built a pipeline system for this task.2 The system consists of basic
components: mention detection, candidate generation, candidate ranking,
NIL classification and NIL clustering. These five components are commonly required for performing Entity Discovery and Linking (EDL) [22].
We add the candidate pruning process after candidate generation to eliminate noisy candidates. In this work, since we want to eliminate the effectof the performance of mention detection, we train and test on the gold
mention data set and start from the candidate generation phase. In order
to simplify the evaluation, we remove the NIL clustering process.
3.1

Candidate Generation

In the candidate generation phase, we need a high-recall candidate list
for each mention. In this phase, recall means the percentage of mentions
that have the correct entity in the candidate list. If the correct entity does
not exist in the candidate list, the ranking process will be in vain. We firstgroup mentions in the source document to handle misspelling, abbreviation, and partial names. For example, the candidate mentions Gretzy and2

We submitted the system to TAC KBP 2014 entity discovery and linking shared
task.

54

S. ZHOU, C. KRUENGKRAI, N. OKAZAKI, AND K. INUI

Wayne Gretzky occur in the same source document, and they likely referto the same entity. If we search candidates by using both of them, the possibility of correct entity appearing in the candidate list of Gretzy could be
increased.
Moreover, we construct a name variation database, SurfaceSet, by extracting entity title-surface pairs from various Wikipedia sources, such as
disambiguation pages, redirection pages, and anchor texts. For example,
we extracted name variations like Barcodes, Toon, mags, magpies, andNewcastle for Newcastle United F.C., a famous England football club.
SurfaceSet contains 548,084 entities and 2,080,491 surfaces.
We process one mention at one time. For each mention, we searchboth the original mention and the same group mentions. We achieved
98.43% recall on the training set. The average number of candidate perlist is 245.
3.2

Candidate Pruning

Note that the initial candidate lists are too noisy because we want to findas many as possible candidates in the previous phase. Ranking document
similarities between source documents and wiki texts is a simple and efficient way to eliminate noisy candidates. According to our preliminary
experiment, we found that Latent Semantic Index (LSI) is superior to
TF/IDF cosine similarity. LSI achieved 97.39% recall on the training set
while TF/IDF got 74.84%. We apply Latent Semantic Index (LSI) to rankeach candidate list and retain the top 50 candidates as the final candidatelist. We use an off-the-shelf tool, gensim [23]. The average number ofcandidates per list is 41.
3.3

Candidate Ranking

In the candidate ranking phase, we formulate the ranking problem similar
to [5, 24]. We generate a score function f (m, ei ) based on features thatextracted from the mention m and a candidate ei . We select a candidate
entity from candidate list E for a mention according to the highest score:e = argmaxei f (m, ei ), ei ∈ E(1)Therefore, the correct entry e for a mention m obtain a higher scorethan all other candidates ê ∈ E, ê 6= e. We use SVMrank [25] with thelinear kernel to handle the optimization problem.

EXPLORING LINGUISTIC FEATURES FOR NAMED ENTITY ...

55

3.4 NIL Classification
We use heuristic rules to determine the final label for a mention. Mentions
are labeled as NIL if there is no candidate in the candidate list or the
ranking score of the top 1 candidate is below a threshold.
4

F EATURE S TUDY

We extract multiple features for candidate ranking. First, we extract basic
features from mention surfaces. In order to explore linguistic information in context, we categorize those linguistic-based features into several
groups.
4.1

Basic Features

We focus on the surface properties of the KB title and the mention surface. The IsAcronym and IsAbbrMatch features [6] capture characteristics of acronyms. For example, given a mention WTO, acronym features can detect World Trade Organization. The SurfaceSimScore andEqualWordNumSurface features [26] calculate how similar is the mention surface to the KB title. The TokenLenInCandidate and CharLenInCandidate [12] count the terms and characters of the KB titles. We also
incorporate other similarity features used in previous work [12, 27], suchas dice coefficient scores and jaccard index scores. We summarize thebasic features in Table 1.
4.2

Linguistic Context Features

We extract linguistic information from both mention source documents
and texts of knowledge base entries (candidates) for disambiguation.
Title appearance
Title appearance features [12] are related with theappearance of a candidate title in the source document, or the appearanceof mentions in candidate texts. For example, if a given mention is the family name of a person, e.g., Daughtry, the title of a candidate, e.g., Chris
Daughtry, may appear in the source document. Similarly, this given mention Daughtry may occur in the text of KB entry Chris Daughtry. Amongthem, a salient feature [12] detects disambiguators in candidate titles, e.g.,magazine in People (magazine) and basketball in Maurice Williams (basketball).

56

S. ZHOU, C. KRUENGKRAI, N. OKAZAKI, AND K. INUI

Table 1. Basic Features of Candidate Ranking Module
FeatureSurfaceSimScore
EqualWordNumSurface
HasQueryGroup
QueryGroupMatch
QueryGroupOverlap
QueryGroupMaxSim
TokenLenInCandidate
CharLenInCandidate
IsAcronym
IsAbbrMatch
DiceTokenScore
DiceToken
JaccardTokenScore
DiceCharacterScore

DiceCharacter
DiceAlignedTokenSocre
DiceAlignedToken
DiceAlignedCharacterSocre
DiceAlignedCharacter

Description
Levenshtein edit distance between the KB title
and the mention surface
Maximum of count of exact matches between
mentions in the same group and the KB title
Whether the KB title belongs to a mention group
Whether the KB title matches any surface in thesame group
Whether a surface in the same group is substringof the KB title, or vice versa
Maximum similarity between the KB title andsurfaces in the same group
Term count in the KB title
Characters count in the KB title
Whether the mention surface is an acronym
Whether the capital character of the KB title
match any surface in the same group
Maximum value of the dice coefficient between
the KB title token set and the surface token set
Whether DiceTokenScore is above 0.9
Maximum value of jaccard index between the
KB title token set and the surface token set
Maximum value of dice coefficient between the
KB title character set and the surface character
set
Whether DiceCharacterScore is above 0.9
Maximum character dice coefficient of left andright aligned token sets
Whether DiceAlignedTokenScore is above 0.9
Maximum character dice coefficient of left andright aligned character sets
Whether DiceAlignedCharacterSocre is above
0.9

Document similarity
We use two measures to compare the text similarity between source documents and KB texts: cosine similarity with
TF/IDF [26] and dice coefficient [27] on tokens. Since the first paragraphs
of KB and text surrounding mention are supposed to be more informative,
we consider using different ranges of source documents and KB texts. We
divide text in a source document into local text (window size = 50 tokens)

EXPLORING LINGUISTIC FEATURES FOR NAMED ENTITY ...

57

and global text (the whole source document), and use the first paragraphand the whole KB text receptively.
Entity mention occurrence
Named entities in mention context are
more salient than common words. This feature is used in [6], which
could capture the count of co-occurring named entities between source
documents and KB texts. For example, for a given mention Obama, thenamed entities White House and United States may appear in both thesource document and the KB text if it refers to the American president
Barack Obama.
Entity fact
The infobox of KB contains important attributes of entries. For example, for entity Apple Inc., we can extract attributes, suchas Founder (Steve Jobs) and CEO (Tim Cook). Therefore we extract facttexts from KB and check whether fact texts are in source documents,
which is inspired by [6]Document topics
Semantic information cannot be detected by simply counting occurrences of tokens, n-grams, and entities. Therefore weuse topic models to discover the implicit topics of source documents and
KB texts. We train LDA (Latent Dirichlet Allocation) model with gensim [23], which provides a fast online LDA model. We treat each KB
entry as one document and use two different corpus for training. Zhang
et al. [11] trained a topic model on the KBP knowledge base, we additionally train another topic model on the latest wikidump.3 The KBP
knowledge base is a partial KB and contains about one third of Wikipedia
entities. We use two similarity measures to check the topic similarity between source documents and KB entries including cosine similarity andHellinger distance. We also generate topics of partial text surrounding
mention as the local topics to compare with using the whole source document (global topics).
Similarity of part-of-speech tokens
We hypothesize that nouns andverbs compared to other type of words could contribute more on disambiguating. Therefore we collect this two type of tokens in context andcalculate cosine similarity with TF/IDF weighting respectively.
Entity type We use entity type matching to detect whether the KB entity type is identical to the mention entity type, which is similar with [6].
For example, the mention St.Andrew is an ORG (Organization) entity inthe first text in Figure 2. The candidate University of St.Andrew (ORG) is3

http://dumps.wikimedia.org/enwiki/20140707/enwiki20140707-pagesarticles.xml.bz2

58

S. ZHOU, C. KRUENGKRAI, N. OKAZAKI, AND K. INUI

more likely than St.Andrew, Scotland (GPE) because of entity type matching. Therefore, we should predict named entity types for both non-NIL
mentions and NIL mentions in the final output results. Since the KBP KB
provides entity type information, we concern that it is more credible topredict non-NIL mention types by using KBP KB labeled type. However
there are almost 64.9% unknown entities in the official KBP KB. It meansthat we need to re-tag remaining unknown entities. Unlike [6], we use there-tag entity types according to our re-tagging results.
Clarke et al. [28] classified unknown type entities based on the infobox class. They also found that matching between infobox classes andentity types approximately has no ambiguity. Unlike [28] classified infobox class using learning method, we resolved around 2370 infobox
classes manually. Our re-tagged result contains four types: PER, ORG,
GPE, and MISC. Table 2 shows the comparison before and after retagging process.
Table 2. Entity types before and after re-tagging
Type
PER
ORG
GPE
UKN
MISC

KBP KB
14.0%
6.8%
14.2%
64.9%
0.0%

5

E VALUATION

5.1

Data Set and Evaluation Metric

Our System
23.5%
12.3%
22.0%
0.0%
42.2%

We use the training data from the 2014 TAC KBP Entity Discovery and
Linking (EDL) track [22]. The TAC data set consists of 5878 mentions
over 158 documents. The statistics of the data set is shown in Table 3. We
use the gold mention query file of the data set.
Our evaluation metric is micro-averaged accuracy, which is used in
TAC KBP 2009 and 2010 entity linking task [13]. The metric is computedby
N umCorrrect(2)
Accuracymicro =N umM entions

EXPLORING LINGUISTIC FEATURES FOR NAMED ENTITY ...

59

Table 3. Statistics of 2014 TAC KBP Data set

NIL
Non-NIL
Total

PER
1819
1390
3209

ORG
591
709
1300

GPE
216
1153
1369

Total
2626
3252
5878

5.2 Experiment
Since we focus on the ranking performance of each group of linguisticbased context features, we compute the accuracy of mentions system resolved. In order to eliminate the effect of feature combination, we add
only one feature group to the basic feature group each time. We performed 5-fold cross-validation on the training set. Table 4 shows microaveraged accuracies of feature addition experiments.
Table 4. Feature additive test results
Feature Group
Basic
Title Appearance
Entity Fact
Entity Mention Occurrence
Document Similarity
Document Similarity (LOCAL)
Document Similarity (GLOBAL)
Document Topic
Document Topic (WIKI)
Document Topic (KBP)
Similarity of POS
Similarity of POS (Noun)
Similarity of POS (Verb)Type
All Features

Non
0.59100.61380.60240.61340.65940.64220.6474
0.63220.62240.62800.62240.6236
0.59860.59080.7330NIL
0.70000.7086
0.66640.76680.77330.74030.7881
0.69120.69120.68800.74200.73640.69700.70300.7454ALL
0.63940.65580.6306
0.68140.70590.68600.70960.65800.65280.65440.6754
0.6736
0.64160.64000.7378In Figure 3, we consider subsets of mentions by entity type. We compare the entity linking performance on three types respectively.
In order to clarify feature effects, we divide features into more finegrained groups, such as local topics (DT WIKI LOC, DT KBP LOC),
global topics (DT WIKI GLO, DT KBP GLO), and document similarity
by using the first paragraph of KB texts (DS CON FIR) or the whole KB

60

S. ZHOU, C. KRUENGKRAI, N. OKAZAKI, AND K. INUI

0.9	
  
0.8	
  
0.7	
  
0.6	
  
0.5	
  
PER-­‐ALL	
  

0.4	
  

ORG-­‐ALL	
  

0.3	
  

GPE-­‐ALL	
  

0.2	
  
0.1	
  

No
un
	
  
PS
_V
er
b	
  
C_
ES
_T
yp
e	
  
C_

PS
	
  
C_

PS
_
C_

_K
BP
	
  

DT

C_

	
  

_W
IK
I	
  

DT
C_

BA
L	
  

C_
DT

C_

DS

_G
LO

AL
	
  

	
  

OC

DS
C_

DS

_L

C_

	
  
EF
C_

EM
O	
  

C_

ic	
  
Ba
s

C_
TA
	
  

0	
  

Fig. 3. Entity linking performance on PER, ORG, GPE entities

texts (DS CON ALL). Table 5 shows the increment of each fine-grainedfeature group to basic features on non-NIL mentions before NIL classification processing, and feature group names are capitalized referring toTable 4.
Moreover, we plan to check whether our current linguistic features
can be used as local methods for global inference methods.Therefore, we
compare the performance of our features with two local methods of previous work: ‘TF/IDF cosine similarity’ and ‘Entity Popularity’. We evaluate the accuracy on the TAC KBP 2014 test data set. Since considering
using local methods to filter out noisy candidates for global inference
methods, we calculate the percentage of correct entities on different positions. The results are shown in Table 6. Acc@1, Acc@5, and Acc@10
mean correct entities on top1 position, within the top5 results, and within
the top10 results.
We only focus on the performance of the in-KB entities because correct entities are provided in the data set. Similar with [20], we retrievea mention with the Freebase search API,4 and we select an entity which
has the highest popular score of all the returned entities. We found that
our features overcome ‘TF/IDF cosine similarity’ and ‘Entity Popularity’in all the three cases.
4

https://developers.google.com/freebase/v1

EXPLORING LINGUISTIC FEATURES FOR NAMED ENTITY ...

61

Table 5. Accuracy increment on non-NIL mentions before NIL classification
Fine-grained Feature GroupC DS LOCALC DS GLOBALC DS CON FIR
C DS CON ALL
C DT
C DT WIKI
C DT KBPC DT WIKI GLOC DT WIKI LOCC DT KBP GLO
C DT KBP LOC
C PS Noun
C PS Verb

Accuracy Increment
0.07360.10440.02140.0726
0.05820.03380.0576
0.0576
0.03440.05340.05100.06580.0150

Table 6. Accuracy at different positions

TF/IDF cosine similarity
Entity Popularity
All Features

6

Acc@1
0.5066
0.37100.8264Acc@50.82040.54530.9418

Acc@100.89100.61240.9492

D ISCUSSION AND F UTURE W ORK

6.1 Overall Feature Effects
Basic features only include features related to surface similarity, which
is not effective enough to find correct entities. Features based on document similarity (both words and part-of-speech levels), named entities
co-occurrence, and document topics contribute the most gains.
Document similarity In both document similarity and document topics, global features are better than local features. Since we leverage measures based on bag-of-words calculation, the larger text of context contains more co-occurring words than the window-size context. Although
we suggest that the first paragraph in the KB is much informative, using
the whole KB text (DS CON ALL) is much better than only using thefirst paragraph (DS CON FIR). We found that, in the KBP KB, several
first paragraphs of KB texts are very short, sometimes only one sentence.
For example, for Jeff Perry (American actor), there is only one sentence,

62

S. ZHOU, C. KRUENGKRAI, N. OKAZAKI, AND K. INUI

“Jeff Perry (born August 16, 1955 in Highland Park, Illinois) is an American character actor.”
We found that around 28.74% entries of the KBP KB contain one
simple sentence in the first paragraph.
Document topics Moreover, based on the results in Table 5, the increment of global topics is more than that of local topics by 0.024 (KBP corpus). Since the distribution of partial document topics is inconsistent with
document topics, global topics can better represent the semantic context
of a mention.
Although the KBP KB contains around one third entities of Wikipedia,
the performance on the KBP KB corpus is better because we use the KBP
KB as the entities database. We found that words of KBP KB topics could
represent source document better than using the Wikipedia corpus forsome entities. For example, Salvador Dali entity is a painter, who is also
known for writing and film. Words of top topics are given by the KBP
LDA corpus of this entity are film, book, album, play. However, words
given by the Wikipedia LDA corpus are Louisiana, disease, species, andso on. The Wikipedia LDA corpus is not well-built, which may also affectthe performance, because we follow an off-the-shelf training process.5
However, from Table 4 one can see that the performance on Wikipedia
corpus is slightly effective on NIL by 0.003.
Similarity of POS tokens
In Table 4, we found that nouns are more
informative than verbs by around 0.3. Nouns contain more information
than verbs because named entities are more salient.
6.2

Feature Effects on Different Entity Types

Figure 3 shows the performance on PER entities is much better than ORG
and GPE entities by more than 20 percentage. It reveals that our features
are biased toward PER entities and ORG and GPE mentions are difficult
to resolve. After the error analysis, we found that simple string matching
linguistic features fail to disambiguate ORG and GPE entities because ofmultiple name variation, especially the confusion between different entity types. For example, city names could be part of sport teams (Orlando
is short for Orlando Magic) and people names could be part of company
names (Disney is short for Walt Disney Company or Walt Disney Animation Studio). Moreover, the amount of PER mentions is two times larger
than the amount of ORG mentions or GPE mentions in our data set.
5

https://radimrehurek.com/gensim/wiki.html

EXPLORING LINGUISTIC FEATURES FOR NAMED ENTITY ...

63

6.3 Future Work
We compared the performance of current features with systems from the2014 EDL Diagnostic task [22]. The accuracy on all mentions of ourcurrent system could beat the median system by 0.5, but we still have ahuge gap with the best system. Even for some top systems from the 2014
EDL workshop [22] , performance on ORG and GPE entities are stillmuch worse than PER entities. It should be an important future work to
discover effective features which can solve ORG and GPE entities better.
Since we use a simple heuristic method to classify non-NIL and NIL
mentions, the accuracy significantly drops after NIL classification process. In future work, we will explore effect features on determining NIL
entities and improve the NIL classification method. Moreover, we plan to
combine linguistic features with link-based methods to further improve
our system.
R EFERENCES
1. Ji, H., Grishman, R., Dang, H.T., Griffitt, K., Ellis, J.: Overview of the tac
2010 knowledge base population track. In: Third Text Analysis Conference(TAC 2010). (2010)
2. Ratinov, L., Roth, D., Downey, D., Anderson, M.: Local and global algorithms for disambiguation to Wikipedia. In: Proceedings of the 49th Annual
Meeting of the Association for Computational Linguistics: Human Language
Technologies-Volume 1. (2011) 1375–1384
3. Erbs, N., Zesch, T., Gurevych, I.: Link discovery: A comprehensive analysis. In: 2011 Fifth IEEE International Conference on Semantic Computing(ICSC), IEEE (2011) 83–86
4. Hoffart, J., Yosef, M.A., Bordino, I., Fürstenau, H., Pinkal, M., Spaniol, M.,
Taneva, B., Thater, S., Weikum, G.: Robust disambiguation of named entities
in text. In: Proceedings of the Conference on Empirical Methods in Natural
Language Processing. (2011) 782–792
5. Bunescu, R.C., Pasca, M.: Using encyclopedic knowledge for named entity
disambiguation. In: Proceedings of EACL. Volume 6. (2006) 9–16
6. Dredze, M., McNamee, P., Rao, D., Gerber, A., Finin, T.: Entity disambiguation for knowledge base population. In: Proceedings of the 23rd International
Conference on Computational Linguistics. (2010) 277–285
7. Cucerzan, S.: Large-scale named entity disambiguation based on Wikipedia
data. In: EMNLP-CoNLL. Volume 7. (2007) 708–716
8. Milne, D., Witten, I.H.: Learning to link with Wikipedia. In: Proceedingsof the 17th ACM Conference on Information and Knowledge Management,
ACM (2008) 509–518

64

S. ZHOU, C. KRUENGKRAI, N. OKAZAKI, AND K. INUI

9. Guo, Z., Barbosa, D.: Robust entity linking via random walks. In: Proceedings of the 23rd ACM International Conference on Conference on Information and Knowledge Management, ACM (2014) 499–508
10. Han, X., Sun, L., Zhao, J.: Collective entity linking in web text: A graphbased method. In: Proceedings of the 34th International ACM SIGIR Conference on Research and Development in Information Retrieval, ACM (2011)
765–774
11. Zhang, W., Sim, Y.C., Su, J., Tan, C.L.: Entity linking with effective acronym
expansion, instance selection, and topic modeling. In: Proceedings of IJCAI.
Volume 2011. (2011) 1909–1914
12. Graus, D., Kenter, T., Bron, M., Meij, E., De Rijke, M.: Context-based entity
linking – University of Amsterdam at TAC 2012. In: Proceedings of theFifth Text Analysis Conference (TAC 2012): November 5–6, 2012, National
Institute of Standards and Technology, Gaithersburg, Maryland, USA. (2012)
13. McNamee, P., Dang, H.T.: Overview of the tac 2009 knowledge base population track. In: Text Analysis Conference (TAC). Volume 17. (2009) 111–113
14. Garcıa, N.F., Fisteus, J.A., Fernández, L.S.: Comparative evaluation of linkbased approaches for candidate ranking in link-to-Wikipedia systems. Journal of Artificial Intelligence Research 49 (2014) 733–773
15. Han, X., Zhao, J.: Named entity disambiguation by leveraging Wikipedia
semantic knowledge. In: Proceedings of the 18th ACM Conference on Information and Knowledge Management, ACM (2009) 215–224
16. Kulkarni, S., Singh, A., Ramakrishnan, G., Chakrabarti, S.: Collective annotation of Wikipedia entities in web text. In: Proceedings of the 15th ACM
SIGKDD International Conference on Knowledge Discovery and Data Mining, ACM (2009) 457–466
17. Guo, Y., Tang, G., Che, W., Liu, T., Li, S.: Hit approaches to entity linking
at tac 2011. In: Proceedings of the Fourth Text Analysis Conference (TAC
2011). (2011)
18. Guo, S., Chang, M.W., Kiciman, E.: To link or not to link? a study on endto-end tweet entity linking. In: HLT-NAACL. (2013) 1020–1030
19. Ferragina, P., Scaiella, U.: Tagme: On-the-fly annotation of short text fragments (by Wikipedia entities). In: Proceedings of the 19th ACM International Conference on Information and Knowledge Management, ACM(2010) 1625–1628
20. Alhelbawy, A., Gaizauskas, R.: Graph ranking for collective named entity
disambiguation. In: Proceedings of the 52nd Annual Meeting of the ACL.
(2014) 75–80
21. Alhelbawy, A., Gaizauskas, R.: Collective named entity disambiguation using graph ranking and clique partitioning approaches. In: Proceedings of
COLING. (2014) 1544–1555
22. Ji, H., Dang, H., Nothman, J., Hachey, B.: Overview of tac-kbp2014 entity
discovery and linking tasks. In: Proceedings of Text Analysis Conference.
(2014)

EXPLORING LINGUISTIC FEATURES FOR NAMED ENTITY ...

65

23. Řehůřek, R., Sojka, P.: Software framework for topic modelling with large
corpora. In: Proceedings of the LREC: Workshop on New Challenges for
NLP Frameworks. (2010) 45–50
24. McNamee, P., Dredze, M., Gerber, A., Garera, N., Finin, T., Mayfield, J., Piatko, C., Rao, D., Yarowsky, D., Dreyer, M.: Hltcoe approaches to knowledge
base population at tac 2009. In: Proceedings of TAC 2009. (2009)25. Joachims, T.: Training linear SVMs in linear time. In: Proceedings of the
12th ACM SIGKDD International Conference on Knowledge Discovery andData Mining, ACM (2006) 217–226
26. Zheng, Z., Li, F., Huang, M., Zhu, X.: Learning to link entities with knowledge base. In: Human Language Technologies: The 2010 Annual Conference of the North American Chapter of the Association for Computational
Linguistics, Association for Computational Linguistics (2010) 483–491
27. Dietz, L., Dalton, J.: Acrossdocument neighborhood expansion: Umass at
tac kbp 2012 entity linking. In: Proceedings of TAC 2012. (2012)
28. Clarke, J., Suleiman, Y.M.G., Murgatroyd, S.Z.D.: Basis technology at tac
2012 entity linking. In: Proceedings of TAC 2012. (2012)

S HUANGSHUANG Z HOUI NUI -O KAZAKI L AB .,G RADUATE S CHOOL OF I NFORMATION S CIENCES ,6-6 A RAMAKI A ZA AOBA , AOBAKU , S ENDAI , M IYAGI 980-8579,
JAPAN
E- MAIL : <SHUANG @ ECEI . TOHOKU . AC . JP>C ANASAI K RUENGKRAII NUI -O KAZAKI L AB .,G RADUATE S CHOOL OF I NFORMATION S CIENCES ,6-6 A RAMAKI A ZA AOBA , AOBAKU , S ENDAI , M IYAGI 980-8579,
JAPAN
E- MAIL : <CANASAI @ ECEI . TOHOKU . AC . JP>N AOAKI O KAZAKII NUI -O KAZAKI L AB .,G RADUATE S CHOOL OF I NFORMATION S CIENCES ,6-6 A RAMAKI A ZA AOBA , AOBAKU , S ENDAI , M IYAGI 980-8579,
JAPAN
E- MAIL : <OKAZAI @ ECEI . TOHOKU . AC . JP>

66

S. ZHOU, C. KRUENGKRAI, N. OKAZAKI, AND K. INUIK ENTARO I NUII NUI -O KAZAKI L AB .,G RADUATE S CHOOL OF I NFORMATION S CIENCES ,6-6 A RAMAKI A ZA AOBA , AOBAKU , S ENDAI , M IYAGI 980-8579,
JAPAN
E- MAIL : <INUI @ ECEI . TOHOKU . AC . JP>