SUMMA at TAC Knowledge Base Population Task 2017
Afonso Mendes#David Nogueira#
Samuel Broscheit#†
Filipe Aleixo#†###
Pedro Balage
Rui Martins
Sebastião Miranda
Mariana S. C. Almeida#†#
Priberam Labs, Alameda D. Afonso Henriques, 41, 2o , 1000-123 Lisboa, Portugal†
Instituto de Telecomunicações, Instituto Superior Técnico, 1049-001 Lisboa, Portugal{amm,david.nogueira,pedro.balage,sebastiao.miranda}@priberam.pt

Abstract
This paper describes the SUMMA system for
the Trilingual Entity Discovery and Linking(EDL) for the TAC 2017 Knowledge Base
Population track. We used an entity recognition based on a LSTM+CRF neural networkand two different approaches for entity linking
disambiguation: a nearest-neighbors searchengine and a distributed representation basedon previous work of Yamada et al. (2017). We
submitted 3 runs for named and nominal EDL
task, across all 3 languages (English, Spanish
and Chinese). Our system scored as first out of
24 teams for named mention linking (NERLC)
and mention clustering (CEAFmC) for English
and as third in the same metrics for Spanish.

1

Introduction

Our submission to the NIST TAC-KBP-20171 is anattempt to apply our ongoing research on knowledge base population within the SUMMA2 project
to TAC shared tasks. The goal of SUMMA isto develop a scalable and extensible media monitoring platform with an extensible automated knowledge base construction and cross-lingual capabilities, thus having a significant overlap with TAC-KBP
tasks. Following last year’s submission, restricted
to English named entity disambiguation, we present
an enhanced version of our Entity Discovery and
Linking (EDL) system adapted to the TAC EDL task,
both for named and nominal entities, and in the threelanguages: English, Spanish and Chinese.
12

https://tac.nist.gov//2017/KBP/
http://www.summa-project.eu/

The paper is organized as follows: Section 2
describes our contribution to the EDL track. Experimental results are reported in Section 3. Section 4
concludes the paper.

2

Entity Discovery and Linking

2.1

Overview and submissions description

Five systems were submitted to the EDL track,
although two of them were discarded as their output
did not correlate with our intentions for such runs.
Our entity linking system submitted in run #1,
summa1, uses an information retrieval NearestNeighbors-assisted rule based system (Amaral etal., 2008) to rank candidates and generate additional features from Wikipedia data. Besides nearestneighbors search engine generated features and priorfeatures, features taken from the co-occurrences
between mentions and candidates in Wikipedia, andcoherence features (existence or absence of links
between Wikipedia articles) are used in the candidates re-ranking steps in this first run.
Runs #4 and #5 follow a different approach fortheir disambiguation step, as they are anchoredin English distributed representations provided by
Yamada et al. (2017) for English and trained representations using the same method for Spanish. A
slightly different deep neural network from the one
described in Yamada et al. (2017) was used in these
two runs. Besides distributed representations, run #4
also receives as input data the nearest-neighbors similarity (search engine generated) features, prior andco-occurrences features used in run #1. The underlying idea of these two runs (#4 and #5) was to assesswhether these distributed representations captured
more relevant information for entity disambiguation
than the co-occurrences and link statistics extracted
from Wikipedia.

CRF

Gating (ENG)

GatingGating (SPA)
Logits (ENG)

2.2

Entity Recognition

2.2.1 Pre-processing of NER datasets
To improve the results for the named entity recognition, we preprocessed the following corpora thatwere used for training: TAC 2015 (train); Ontonotes(train); TAC 2016 (dev).
OntoNotes was processed to achieve congruencewith the TAC dataset, namely in the name tagging andspan boundaries. Specifically, we used the domainsof broadcast news, the newswire and the web data;
we shortened span boundaries and mapped spanswith ”Nationality, or Religious or Political Organization” (NORP) tag to GPE, LOC, ORG or no tag (tagremoved).
2.2.2 Named Entity Recognition
The used system is based on the Bi-LSTM+CRF
algorithm described by Ma and Hovy (2016)3 ,where we used GloVe (Pennington et al., 2014)
300 dimensional pre-trained embeddings for English
and we used the word2vec implementation in
Gensim (Řehůřek and Sojka, 2010) to train wordembeddings on the Spanish Wikipedia, where theanchors have been replaced by the KBID, asdescribed by Yamada et al. (2017).
Regarding ensemble architectures, for English weused an ensemble model from 10 runs with randomly
initialized weights; for Spanish, we train a multilingual model using both Spanish and English train data;
and for the Chinese submission, we used the Stanford
CoreNLP (Manning et al., 2014) as the NER system.
The Spanish model trained only with Spanish TAC
dataset reported low scores mainly due the scarceamount of train data available for the language. Ourarchitecture might be described as a jointly trained
ensemble and it is illustrated in Figure 1.
In this architecture, English and Spanish submodels share the same architecture. A gating layer
is then trained with the goal of linearly combining
the logits obtained from the English and Spanish submodels. The gating layer takes as input the list of3

Logits (SPA)

Using as starting point the TensorFlow implementation inhttps://github.com/guillaumegenthial/sequence tagging

ENG
sub-model

SPA
sub-model

Input data

Figure 1: Joint ensemble architecture used for Spanish
NER.

pre-trained word embeddings for each sentence andaverages it, in order to obtain sentence representations. We take the dot product of these sentences andapply a softmax layer that is subsequently used tolinearly combine the logits.
The logits obtained from the linear combination
are then given as input to a CRF layer, such that asingle transitions tensor is learned.
The training procedure is divided into 4 different
stages, alternating between English and Spanish andwith and without gating between the languages.
2.2.3

Nominal Entity Recognition

For nominal mentions we used a lexicon-based
system where the lexicon was constructed fromtraining data and compiled lists. Such lexicon was
divided between two extensive lists, one containing
possible nominal PER surface mentions and otherwith GPE, FAC, LOC and ORG surface mentions. In
order to create the first list, we compiled peoplerelated nominal mentions concerning professions,family relations and adjectives. Regarding the second
list, it contained man-made locations pointing to FAC,
natural locations to LOC, locations with some kindof government to GPE and companies or groups ofpeople to ORG. Those lists were the core of thenominal detection system. After being identified
in the text, the mentions enter a rule-based system
where they are awarded/withdrawn points if theymeet specific parameters. The rules take into accountthe context (previous and following words) as wellas part-of-speech (POS) and dependency parsing

features, extracted using TurboParser4 . More points
were awarded if TurboParser classified the word as
a noun and deducted if it was classified as a verb,
for instance. The same principle was applied to thedependency relation that the word in question had in
the sentence, once again awarding more points if theword was the subject of the sentence. In the case of
the word being identified in one of the lists as a FAC
or ORG, the article used immediately before wastaken into consideration, awarding more points if thearticle was definite and deducting if it was indefinite.
Such system was only used in English and Spanish
submission, whereas no Chinese nominal detection
was performed.
2.2.4 NER post-processing step
As we used the same previously described entity
recognition system on each submission, summa1,
summa4 and summa5 share the same mentions
list as input for the entity linking step. As a postprocessing step, we apply a string matching procedure to capture named mentions that were not recognized by the entity recognition system. In particular,
we extract mentions with the exact same surface as
those previously detected in the document. These
new mentions are then tagged with the types of theold ones, according to a voting procedure that isbiased towards the PER label. Later, in Section 2.3,
some of the mentions types are also reassigned inorder to promote label agreement between mentionsin the same mention cluster (after both the coreference and the linking steps).
2.3

Entity Linking

In this section, we describe the multiple approaches
undertaken towards named and nominal entity disambiguation. Run #1, summa1, is anchored in anearest-neighbors-assisted ruled-based entity linking
system, of which an earlier version was submitted to
TAC KBP EDL 2016 (Paikens et al., 2017). The
mentions detected as reported in Section 2.2 arelinked to KB entries according to the rule-based
strategy that will be described in the following
sections and summarized in Algorithm 1.
Run #4, summa4, of which steps are summarized in Algorithm 2, shares the same initialsteps with summa1, up to the extraction of the4

https://github.com/Priberam/TurboParser

Algorithm 1 summa1 Entity Linking System
1: High-precision sub-string match mention coreference
2: Candidate generation
3: Candidate rank step #0: information retrieval engine(kNN algorithm) + prior statistics
accounting for cooccurrences between all mentions candidates
5: Candidate re-rank step #2: accounting for coherence
6: Attempt to fix NIL mentions with edit distancethreshold
7: Nominal mentions disambiguation
8: Global NIL clustering and cross-document coherence
4: Candidate re-rank step #1:

following features: nearest-neighbors similarity
features, Wikipedia prior features and features takenfrom the co-occurrences between all mention candidates. It then follows a different approach for thenamed disambiguation step. Following recent work
towards jointly mapping words and entities into thesame continuous vector space, we used the English
distributed representations provided by Yamada et al.
(2017) for English and trained Spanish representations using the same method. However, our ranking
neural network architecture differs from Yamada etal. (2017). The differences will be outlined further inthe disambiguation section. The nominal resolution
and post-processing step is the same as in summa1.
Algorithm 2 summa4 Entity Linking System
1: High-precision sub-string match mention coreference
2: Candidate generation
3: Features generation: information retrieval engine(KNN algorithm) + prior statistics + mentions candidates co-occurrences
4: Distributed representation neural network disambiguation
5: Nominal mentions disambiguation
6: Global NIL clustering and cross-document coherence

Run #5, summa5, only shares the initial highprecision sub-string match mention coreference step
(mention clustering) and the final steps. Its candidate generation differs from summa4 in the way
the candidate list is expanded to reach optimal
coverage. The disambiguation step is similar to theone used in summa4, with the only difference, thatno search engine features are used but a document
level coreference feature and a feature describing
the level of fuzziness that was used to retrieve acandidate. As in summa4, the nominal resolution
and post-processing step is the same as in summa1.
Summa5 can be described in the Algorithm 3.
Algorithm 3 summa5 Entity Linking System
1: High-precision sub-string match mention coreference
2: Candidate generation (different from run #1 and #4)
3: Distributed representation neural network disambigua-tion
4: Nominal mentions disambiguation
5: Global NIL clustering and cross-document coherence

Entity linking indexes In the search-engineassisted runs, due to the necessity of linking recognized mentions to known entities, such task is intertwined with an auxiliary procedure, which involves
building a database, to be queried in run-time, inwhich all linkable entities must be stored. For every
language, we generated two information-retrieval
indexes using Wikipedia as the source of information. The first index stores the content of each entity
Wikipedia page as a bag of words, lemmas anddetected entities in an inverted index. A second index
is created, using the anchors information. Wikipedia
anchors are used to discover alternative names toentities, to extract conditional probabilities of entities given those names and to derive co-occurrence
models of mention/entity pairs. Accordingly, eachrecord from such index corresponds to a uniquemention surface, and stores the information of all
the named entities that were linked from its anchor’s
occurrences, as for example, p(e|m), i.e., probabilityof an entity given a mention m and the list of cooccurrent entities with each pair mention/entity.
Mention coreference For each mention, we
perform a high-precision coreference step at the
document level by linking all the surface mentions
which are substrings of other mentions’ forms. To
preserve agreement within the coreference clusters,
we heuristically reassign some entity types with avoting strategy.
Candidate generation Two candidate generation
mechanisms were implemented: one for summa1
and summa4 and another for summa5. In summa1
and summa4, for each mention, the candidates are
generated using the less ambiguous mention (definedas the one with the largest span) in the corresponding

coreference cluster. Then, the candidate generation is performed based on the anchors’ statistics
in Wikipedia. In addition, for mentions with fewer
candidates (less than 50), we also consider as candidates the entities whose titles have all the words ofthe query mention. If even after such procedure,
the number of candidates is less than 10, the searchfor candidate entities is performed in an alternate
mention index, in case of existence (i.e., anotherindex from other language that shares multiple entity
surface forms and/or the same alphabet).
In summa5, we only query the anchor index forthe candidate generation. First, we expand the candidate list by querying the index with increasingly
fuzzier searches on the anchor strings. The candidate
set is then augmented with a second round of queries,
where we restrict the set of possible candidates tothe highest ranked candidates from all mentions fromthe current document. The score for ranking thecandidates is computed from the normalized TF-IDF,
boosted by the log(#(e|m)).
The criterion for expanding the candidate list withincreasingly fuzzier searches is the minimum numberof total inlinks, i.e. #(∗|m), for the set of candidates
for a mention. This hyperparameter was tuned on
training data to reach maximal coverage and was set
to 400. The fuzziness level of a query are: full string
match with an anchor, partial token overlap, partial
token overlap with edit distance 1 and prefix length2.
Candidate ranking Let ci,k be the k th candidateof mention mi and ssearch (ci,k , mi ) be the scoreof a nearest-neighbors search engine procedure thatreflects the similarity of the mention’s document withthe text of candidate ci,k composed by its Wikipedia
title and body. In the 3rd step of Algorithm 1, thecandidates of each mention mi are initially sorted
according to this ranking score ssearch (ci,k , mi ).
A model is applied, and a score expressed bysmodel1 (ci,k , mi ) =nXθj ∗ ci,k j

j=1

is obtained, in which ci,k j are ci,k ’s features generated by the polynomial expansion of the nearestneighbors similarity features and prior features, suchas probability of an entity given a mention, and θi arethe feature weights, trained with a pairwise approach,using SVM-rank (Joachims, 2002). For such training,a file with a list of training queries is generated (onequery per mention), in which each line features amention candidate, with its features and a target
value, 1 if that candidate matches the gold one, and 0
otherwise.
Afterwards, a co-occurrence feature is computedfor the top-ranked candidates for all mentions.
Mention candidates co-occurrence feature values will
be greater whenever those entities co-occur withmore mentions and smaller the more ambiguous
those mentions are (mentions with a higher numberof candidates).
In the 4th step, another model is applied and a newscore, smodel2 (ci,k ), with the same expression as theone in the previous step is obtained, but in this casewith the additional co-occurrence feature. After thereorder by the score smodel2 , one last reorder step isapplied, accounting for coherence, as discussed in
the next section.
Coherence re-rank Contrary to other state-of-theart entity linking methods that favor solutions inwhich the entities of a same document are relatedwith each other and that consider all possible combination of mentions candidates (being therefore, NP
hard, (Kulkarni et al., 2009)), prior work typicallyrelax the general collective formulation either byusing continuous formulations (Kulkarni et al., 2009)or by identifying sets of mentions or entities that are
somehow involved in a semantic relation (Hoffart etal., 2011; Ratinov et al., 2011; Sil et al., 2015; Panet al., 2015) to tackle this problem of complexity. In
this step we focus on the top 10 candidates obtained
from the previous step and re-rank them to favorcoherence. Our envisaged coherence model resolves
each mention independently. To achieve coherence,the score of a mention’s candidate is influenced byits coherence with all the candidates of the othermentions in the text:

scoherence (ci,k , mi ) =X sc (ci,k , cj,l )
j6=i,l

|Cj |

,

(1)

where Cj is candidate list of mention mj andsc (ci,k , cj,l ) is a score that accounts for the coherence between the candidate under evaluation (ci,k )and the lth candidate of other mention mj (cj,l ), andwhich is given by:sc (ci,k , cj,l ) =(1 + pkj,l ,1k
2 + pj,lci,k , cj,l share a linkotherwise,(2)
where pj,l is the position of candidate cj,l according
to the previous ranking score and k is a constant that
represents the number of candidates considered forcoherence. This coherence score was empiricallydesigned to consider both coherence (as the existence or absence of a link) and information regarding
previous candidate order.
Our coherence model, in Equation 1, is similar
with the model that was independently proposed by
Globerson et al. (2016).
Distributed representations neural network
disambiguation Our implemented architectureuses a simple multi-layer perceptron (MLP) classifier
with the learned representations, both from the text
and from candidate entities, as features, composedby a relu hidden layer with dropout, followed bya softmax output layer. Contrary to Yamada et al.
(2017), we also add context features and create anintermediate representation of the external featureswith a hidden layer. Figure 2 portraits a graphicalrepresentation of how these features are concatenatedto form the candidate representation.
Candidate Representation

Text
Representation

EntityContext
Embedding Representation

MLP Layer

MentionFeatures

Figure 2: Candidate representation for the neural network
disambiguation

For summa4 and summa5 the network is fed
with• the text representations,• the entity representations,• the dot product between the text representations
and the entity representations,• the context representations.
For summa4 we add the following intermediate
representations of the features from the previous step:• NN-search-engine features,• prior features,• co-occurrence features.
For summa5 we add the following features:• Indicator, if this candidate was ranked first prior
to the disambiguation for any other mention
from this document,• the level of fuzziness that was used to generate
this candidate,• prior features,• character bigram overlap between the mention
and the prefix and the suffix of the anchor.
NIL disambiguation with edit distance threshold
After the named entity disambiguation, for every
mention that was not linked to any KBID, we
generate a new set of candidates for such mentions,where their surface is compared with the other disambiguated named mention surfaces, their entities titles
and those entities links’ titles in the KB. If the Levenshtein distance between the unlinked mention surface
and those strings is smaller than 2, the corresponding
entities get through to the next phase, in which thesementions are disambiguated in the exact same steps
described in Algorithm 1, up to the co-occurrences
re-rank step (steps 3 and 4). During this procedure,
previous disambiguated mentions are locked andtheir selected KBIDs are not allowed to change.
Nominal mentions disambiguation The nominal
resolution is a step performed after the named
mention resolution. Two baseline approaches weredeveloped for nominal entity linking: the firstone, applied in summa4 and summa5, with therationale that a nominal mention is followed by aprevious mentioned named entity, aims at connecting
a nominal mention to the left-closest disambiguatednamed mention entity that shares the same NER tag.

The other baseline approach, implemented insumma1 follows the same procedure used in thenamed resolution (and in the named NIL disambiguation), but restricts the set of possible candidates tothe set of disambiguated named mentions entities
and their links’ entities (the procedure is the samedescribed in the previous section).
Global NIL clustering and cross-document coherence Finally, the last step builds on top of the last
coherence step to promote a new type of coherence
that works at a corpora level. The underlying idea of
this step is to promote coherence along the entities
that co-occurred (with the same mention surface +candidate pair) in different documents.
Let, for each mention mi , D(mi ) be the set ofthe entities to which the other mentions (mj6=i ) in
the document link to. For each entity ei,k to which
the surface of mention mi links to in the full corpus,
let C(ei,k , mi ) be the set of entities that co-occur indocuments where the surface form of mi connects to
eik . We define the cross-document coherence scoreasscdc (ei,k , mi ) = J(D(mi ), C(ei,k , mi )),(3)

where J(.) is the Jaccard similarity:
J(A, B) =

A∩B
.
A∪B

Each mention mi is finally linked to the entity ei,k∗with the highest cross-document coherence score, in
Equation 3.

3

Results

3.1

Entity Recognition evaluation

We evaluated the impact of using available datasets
for NER in TAC 2016 test set. The results reported
in Table 1 show that the TAC train data alone could
not achieve a great performance at NER and NERC
F-scores. The usage of TAC Ontonotes and ACE
improves the results as they add more examples tothe training step.
We believe the access to more datasets is a crucial
factor to improve the NER and NERC scores on TAC.
The best NER system from TAC 2016, USTC team(Liu et al., 2016), reported they had access to an
in-house built dataset which led to their top performance.

Table 1: Impact of the dataset for English named entity
recognition in TAC 2016

Dataset
TAC+ Ontonotes
+ ACE

NER
83.37
87.35
87.26

NERC
79.04
83.82
83.97

For Spanish, the same was verified. Table 2
compares two approaches: one using a simple
LSTM+CRF model and another using our joint
ensemble approach reported in Section 2.2.2. Our
joint ensemble approach allowed us to use theEnglish dataset which improved the NER and NERC
F-scores.
Table 2: Impact of the approach and dataset for Spanish
named entity recognition in TAC 2016

Approach
SimpleJoint ensemble

3.2

DatasetNER
TAC Spanish 82.92
TAC Spanish +TAC English + 85.58
Ontonotes

NERC
79.51
82.69

Entity Linking evaluation

Step ablation Regarding the Nearest-Neighborsassisted ruled-based Entity Linking system,
submitted as summa1, we present the contributionof each step to its performance in Table 3. Relevant
metrics are reported with the ENG and NAM filters,
at the end of steps 3, 4, 5 and 8. Every step led toan improvement across all metrics, which offers acumulative noticeable improvement.
As it can be seen, the best performance is achievedin the final step, with a cumulative improvement
across all steps of 2.4% with regard to mention
detection, type classification and linking (NERLC),
2.0% regarding document entity tagging (KBIDs)
and 2.9% with regard to clustering metrics: detection
and clustering (CEAFm) and detection, clustering andtype classification (CEAFmC). The biggest improvements were due to 1) the SVM re-rank with theco-occurrence features, contributing with 1% F1
improvement to the entity linking metrics, as theco-occurrences prove to be relevant for the disambiguation; and 2) the NIL clustering and corporalevel coherence, contributing with around 1.7% F1

improvement to the clustering metrics, as in thisstep NIL mentions are clustered together and somementions are re-linked to other entities that havehigher cross-document coherence scores, which leads
to better clusters.
Feature ablation Contrary to Yamada et al. (2017),
in the system represented by summa4, we added
context representations and its contribution to theperformance is presented in Table 4. Such results
prove that the context features clearly contribute positively for the linking and clustering performance.
Table 5 evaluates the contribution of each setof external features to the summa4 system performance in the TAC KBP ELD 2016 evaluation corpus.
It is worth noticing that the difference between
training and testing with no external features andwith all features is less or equal to 0.6% in almost
all metrics (with the exception of the KBIDs metric),
which points to the fact that the distributed representations of the text, entities and context alreadyprovide much of the knowledge necessary for thedisambiguation. The best result is obtained with justprior and co-occurrence features, as opposed to using
prior, co-occurrence and also the nearest-neighbors
similarity features, which points to the fact that fromthe set of external features, they are the most relevant
ones and that when training with all those externalfeatures, the system over-fitted to the training data.
Nominal resolution Our contribution to thenominal resolution is small and the results are not
up to par with the ones in the named resolution.
However, experiments with nominal resolution show
us that with the nearest-neighbors search engine
system we are able to attain circa 0.92 recall F1 score,using the set composed by the top-3 ranked candidates retrieved for all the mentions in the document.
It is yet to be found a way to leverage such high recall
to improve precision without reducing too much therecall, as it happens in both our baseline approaches,
where we attained around 0.25 both in precision andrecall, with the ENG-NOM filters. Future work may
involve experimenting with distributed representations of text and entities for nominal resolution.
Official scores For TAC KBP EDL 2017 we
submitted three runs, as previously described.
Regarding Monolingual English EDL (Table 6), ourTable 3: Step ablation in in Nearest-Neighbors-assisted ruled-based EL pipeline system (summa1’s system), for the
TAC KBP EDL 2016 evaluation corpus, with ENG-NAM filter.

first svm-rankwith nearest-neighbors
similarity features (step-3)
second svm-rank withnearest-neighbors similarity
and co-occurrence features (step-4)
document-level
coherence (step-5)
NIL clustering andcorpora-level coherence (step-8)

NER

NERC

NERLC

KBIDs

CEAFm

CEAFmC

0.887

0.855

0.771

0.768

0.822

0.795

0.887

0.855

0.780

0.784

0.831

0.804

0.887

0.855

0.783

0.788

0.834

0.807

0.887

0.856

0.795

0.798

0.851

0.824

Table 4: Impact of context representations in MLP classifier disambiguation (summa4’s system), for the TAC KBPEDL 2016 evaluation corpus, with ENG-NAM filter, using no external usage of features.

without context representations
with context representations

NER
0.887
0.887

NERC
0.856
0.856

NERLC
0.774
0.788

KBIDs
0.804
0.817

CEAFm
0.829
0.842

CEAFmC
0.802
0.815

Table 5: External feature ablation in MLP classifier disambiguation (summa4’s system), for the TAC KBP EDL 2016
evaluation corpus, with ENG-NAM filter.

no external features
just prior and co-occurrence featuresprior, nearest-neighbors similarity
and co-occurrence features

NER
0.887
0.887

NERC
0.856
0.856

NERLC
0.788
0.795

KBIDs
0.817
0.839

CEAFm
0.842
0.852

CEAFmC
0.815
0.825

0.887

0.856

0.794

0.838

0.848

0.820

NERC placed us in the 6th place, with 0.784 F1 score.
We then achieved 0.653 (5th place) for the entity
linking NERLC metric and 0.674 (4th place) for clustering CEAFmC metric. Spanish results (Table 8) aresimilar: 0.750 F1 score for NERC (6th place), 0.594
F1 score for NERLC (also 6th place) and 0.619 F1score CEAFmC metric (5th place).
More impressive results were attained with the
NAM filter, specifically for Monolingual English
EDL (NAM). Although we start from a 4th place in
the NERC metric with 0.861 F1 score (best score was
0.873), we were able to achieve the 1st place both inentity linking NERLC metric and clustering CEAFmC
metric, with 0.794 and 0.831 F1 scores, respectively(Table 7). In the Monolingual Spanish EDL (NAM),
we rank 8th place in the NERC metric with 0.816 F1score (best score was 0.873) and achieved 3st place
both in entity linking NERLC metric and clustering

CEAFmC metric, with 0.745 and 0.788 F1 scores,
respectively (Table 9).

4Conclusions

This paper described the contribution of SUMMA to
the NIST TAC-KBP 2017. In this second year, we
competed in the EDL track. We developed work
on top of our TAC KBP submission of last year,
where our main contributions to the track were our cooccurrence model and coherence steps, both intra andinter-document, in which the latter coherence scorefavors agreement between bags-of-entities along a
corpus in an original approach.
We also submitted a language independent system
to the EDL track, and although we still have a gap
to close in NER and NERC F1-scores comparing withthe best teams, we successfully managed to obtainthe 1st position in English Named disambiguation

Table 6: English results on the TAC-KBP EDL 2017 test dataRunsumma1summa4summa5

NER
0.819
0.819
0.820

NERC
0.782
0.782
0.784

NERLC
0.647
0.653
0.597

KBIDs
0.737
0.748
0.720

CEAFm
0.692
0.695
0.676

CEAFmC
0.670
0.674
0.655

Table 7: English results with NAM filter on the TAC-KBP EDL 2017 test dataRunsumma1summa4summa5

NER
0.885
0.885
0.887

NERC
0.859
0.859
0.861

NERLC
0.784
0.794
0.716

KBIDs
0.773
0.791
0.757

CEAFm
0.849
0.853
0.831

CEAFmC
0.827
0.831
0.809

Table 8: Spanish results on the TAC-KBP EDL 2017 test dataRunsumma1summa5summa4

NER

NERC

NERLC

KBIDs

CEAFm

CEAFmC

0.801

0.750

0.590

0.696

0.646

0.619

0.801

0.750

0.594

0.709

0.644

0.618

Table 9: Spanish results with NAM filter on the TAC-KBP EDL 2017 test dataRunsumma1summa5summa4

NER

NERC

NERLC

KBIDs

CEAFm

CEAFmC

0.857

0.816

0.736

0.737

0.819

0.784

0.857

0.816

0.745

0.755

0.821

0.788

both in NERLC and CEAFmC evaluation metrics and3rd position in Spanish Named disambiguation, alsoin NERLC and CEAFmC metrics.

Acknowledgments
This work was partially supported by the
SUMMA project – this project has received
funding from the European Union’s Horizon
2020 research and innovation programme under
grant agreement No 688139, by Fundação paraa Ciência e Tecnologia (FCT) through contract
UID/EEA/50008/2013, through the LearnBig project(PTDC/EEI-SII/7092/2014), and the GoLocal project(grant CMUPERI/TIC/0046/2014).

References
Carlos Amaral, Adán Cassan, Helena Figueira, André
Martins, Afonso Mendes, Pedro Mendes, José Pina, andCláudia Pinto. 2008. Priberam’s question answeringsystem in qa@ clef 2008. In Workshop of the CrossLanguage Evaluation Forum for European Languages,
pages 337–344. Springer.
Amir Globerson, Nevena Lazic, Soumen Chakrabarti,
Amarnag Subramanya, Michael Ringgaard, and Fernando
Pereira. 2016. Collective entity resolution with multifocal attention. arXiv preprint arXiv:1705.02494.
Johannes Hoffart, Mohamed Amir Yosef, Ilaria Bordino,
Hagen Fürstenau, Manfred Pinkal, Marc Spaniol, Bilyana
Taneva, Stefan Thater, and Gerhard Weikum. 2011.
Robust disambiguation of named entities in text. In
Proceedings of the 2011 Conference on Empirical
Methods in Natural Language Processing, Edinburgh,
Scotland, U.K., 27–29 July 2011, pages 782–792.
Thorsten Joachims. 2002. Optimizing search engines
using clickthrough data. In Proceedings of the eighth
ACM SIGKDD international conference on Knowledge
discovery and data mining, pages 133–142. ACM.
Sayali Kulkarni, Amit Singh, Ganesh Ramakrishnan, andSoumen Chakrabarti. 2009. Collective annotation of
wikipedia entities in web text. In Proceedings of the 15th
ACM SIGKDD international conference on Knowledge
discovery and data mining, pages 457–466.
Dan Liu, Wei Lin, Shiliang Zhang, Si Wei, and Hui Jiang.
2016. Neural networks models for entity discovery andlinking. CoRR, abs/1611.03558.
Xuezhe Ma and Eduard Hovy. 2016. End-to-end
Sequence Labeling via Bi-directional LSTM-CNNs-CRF.
arXiv preprint arXiv:1603.01354.
Christopher D. Manning, Mihai Surdeanu, John Bauer,
Jenny Finkel, Steven J. Bethard, and David McClosky.
2014. The Stanford CoreNLP natural language processing
toolkit. In Association for Computational Linguistics(ACL) System Demonstrations, pages 55–60.
Peteris Paikens, Guntis Barzdins, Afonso Mendes, Daniel
Ferreira, Samuel Broscheit, Mariana S. C. Almeida,
Sebastião Miranda, David Nogueira, Pedro Balage, andAndré F. T. Martins. 2017. Summa at tac knowledge base
population task 2016. In Proceedings of the Text Analysis Conference -TAC, pages 1–9, Gaithersburg, Maryland
USA.
Xiaoman Pan, Taylor Cassidy, Ulf Hermjakob, Heng Ji,
and Kevin Knight. 2015. Unsupervised entity linking
with abstract meaning representation. In Proceedings ofthe 2015 Conference of the North American Chapter ofthe Association for Computational Linguistics–Human
Language Technologies.
Jeffrey Pennington, Richard Socher, and Christopher
Manning. 2014. GloVe: Global Vectors for Word Representation.
Lev Ratinov, Dan Roth, Doug Downey, and Mike
Anderson. 2011. Local and global algorithms for disambiguation to wikipedia. In Proceedings of the 49th Annual
Meeting of the Association for Computational Linguistics: Human Language Technologies - Volume 1, HLT ’11,
pages 1375–1384, Stroudsburg, PA, USA. Association for
Computational Linguistics.
Radim Řehůřek and Petr Sojka. 2010. Software
Framework for Topic Modelling with Large Corpora.
In Proceedings of the LREC 2010 Workshop on
New Challenges for NLP Frameworks, pages 45–50,
Valletta, Malta, May. ELRA. http://is.muni.cz/
publication/884893/en.
Avirup Sil, Georgiana Dinu, and Radu Florian. 2015. The
ibm systems for trilingual entity discovery and linking at
tac 2015. In Proc. Text Analysis Conference (TAC2015).
Ikuya Yamada, Hiroyuki Shindo, Hideaki Takeda, andYoshiyasu Takefuji. 2017. Learning Distributed Representations of Texts and Entities from Knowledge Base.
ArXiv e-prints, May.