Named Entity Sequence Classification
Mahdi Namazifar

arXiv:1712.02316v1 [] 6 Dec 2017

Twitter Inc.

Abstract
Named Entity Recognition (NER) aims at locating and classifying named
entities in text. In some use cases of NER, including cases where detected
named entities are used in creating content recommendations, it is crucial to
have a reliable confidence level for the detected named entities. In this work
we study the problem of finding confidence levels for detected named entities.
We refer to this problem as Named Entity Sequence Classification (NESC).
We frame NESC as a binary classification problem and we use NER as well as
recurrent neural networks to find the probability of candidate named entity
is a real named entity. We apply this approach to Tweet texts and we show
how we could find named entities with high confidence levels from Tweets.
1. Introduction
In Named Entity Recognition (NER) the goal is to locate and classify
named entities (defined as physical or abstract objects that can be expressed
with proper nouns) in a given text. NER as an area of Natural Language
Processing (NLP) has been studied extensively. A survey of studies of NER
based on classical NLP approaches can be found in [1]. NER approaches
based on deep learning have also been frequently reported in the NLP literature [2, 3]. These approaches generally rely on pre-trained word embeddings
as well as sequence modeling techniques based on Recurrent Neural Networks
(RNNs) or Temporal Convolutions.
NER for Tweets has also been studied in numerous studies [4, 5, 6, 7]. Due
to the limit on the number of characters of a Tweet, heavy use of slangs and
emojis, lack of proper capitalization, as well as the informal style of writing,
detecting named entities in Tweet text is significantly more challenging than
other types of text (News, books, web page, etc.).

Preprint submitted to Journal Name

December 7, 2017

Our focus in this work is on detecting named entities in Tweets. The
architecture that we use for our NER model is somewhat similar to the
architecture proposed in [8] in that we use bidirectional LSTMs and Conditional Random Fields (CRF) to tag Tweets tokens with named entity labels.
We show that our trained NER model performs quite well relative to the
standard open source NER solution by the Stanford NLP group [9].
One of the applications of NER on Tweets for Twitter is recommending
content to users. Some example content recommendation use cases where
NER is valuable include notifying the user that their network is Tweeting
about a specific named entity, or grouping Tweets about a certain named
entity to show the users. It is important to note that for recommending content it is crucial to have very high confidence in the detected named entities
to be actually true named entities. Unfortunately it is not straightforward
to measure confidence on a sequence of tokens that is detected by our NER
model (which is a sequence tagging based approach) being a named entity.
In fact none of the studies that we found on sequence tagging would address
this requirement, and that is the motivation behind the problem that we call
Named Entity Sequence Classification (NESC).
We define NESC as follows: given a text and a sequence of tokens in that
text, determine if the sequence of tokens is a named entity. The idea is that
NESC as a binary classification problem would provide the probability of the
sequence of tokens (named entity candidate) being in fact a named entity.
For the NESC model we use the output of the trained bidirectional LSTM
of the NER model for a context window around the candidate named entity
(sequence of tokens that we want to classify as a named entity or not) and
we train another LSTM for the binary classification problem of NESC.
It should be mentioned that this binary classification of sub-sequences of
a sequence is not unique to named entity recognition and could be applied
to any sequence tagging problem where consecutive elements of the sequence
could constitute a tag of interest.
In the rest of the this paper, we first present our NER model for Tweets.
Next we discuss the NESC problem definition, our proposed model architecture for NESC, and also details on how to train the proposed NESC model.
Finally we show some experimental results on performance of our NER and
NESC models.

2

2. NER for Tweets
By definition, NER not only tries to locate named entities in text, but
also tries to classify the detected named entities to a pre-determined set of
entity types. Different NER models have different sets of possible entity
types. In this work for the set of possible entity types that are covered by
our NER model, we adhered to the following types: Person, Place, Product,
Organization, and Other ; and we use the Inside–Outside–Beginning (IOB)
format [10] for entity boundaries.
In order to build an NER model for Tweets we follow the general model
architecture proposed by many authors [11, 12, 7] that includes pre-trained
word embeddings for Tweet tokens followed by a recurrent neural network (a
bidirectional LSTM in our case) and a fully dense layer followed by a softmax
layer that produce a discrete probability distribution for the possible labels
of each Tweet token.
Tweet texts are first tokenized using a Twitter internal text processing
tool which is a heuristic-based text tokenizer for tens of languages. Each
token then is vectorized using our pre-trained word embeddings. Our word
embeddings are 200-dimensional vectors that are trained using GloVe [13]
on over 1 billion Tweets. Other than dense word embeddings, the vector
representing each token also includes some sparse variables in the form of 2
one-hot vectors. The first one-hot vector indicates whether the token is one
of the special characters (%, /, ., !, ?, . . . ), a hashtag, an @handle, whether
it’s first character is capitalized, or the entire token is capitalized 1 . This
one-hot vector has 36 dimensions. The other one-hot vector specifies the
Part of Speech (POS) tag associated with each of the tokens. These POS
tags are also provided by Twitter’s internal text processing tools, and each
token can get one of the 17 possible POS tags; and as a result this one-hot
vector has 17 dimensions. For the non-zero value of the one-hot vectors we
use 0.1 so that this number is in the same range as the values in the dense
word embedding vectors. Figure 1 depicts the parts of a vectorized token.
Vectorized Tweet tokens next go through our NER model that has the
architecture that is shown in Figure 2. As one could see from the figure
vectorized tokens go through a bidirectional LSTM with dropout and the
output goes through a fully connected layer and then a softmax layer to
1

If the entire token is capitalized, the value of the indicator for first character being
capitalized is 0.

3

Figure 1: Vectorized token

create 11 dimensional probability distributions for each token. These 11
dimensions correspond to beginning and inside of the 5 entity type tags plus
the one not-an-entity tag. Next these probabilities go through a Conditional
Random Field (CRF) which learns the correct order of entity labels and the
result is the final NER labels for each of the tokens. For instance it learns
that not-an-entity labels most likely is not followed by a label for inside an
entity label (e.g., I-person).
3. NESC
3.1. Problem Definition
The output of the NER model discussed in the previous section for each
token provides a label along with a probability that is given by the softmax
layer to that label. These scores however cannot directly be used, for example
in a multiplicative way, to get a confidence for multi-token entities. As an
example if we run sentence “I love San Francisco” through the NER model
the output of the softmax layer (probability of each label for each token) are
shown in Table 1.
From the labels in Table 1 we can see that the model predicts that ”San
Francisco” is a place, but it is not straightforward to calculate the likelihood
of “San Francisco” being an entity from the softmax probabilities provided for
the token labels. The reason for that is that these probabilities are calculated
using the entire context of the text and all of the probabilities of tokens and
their labels are correlated.

4

Figure 2: NER model architecture

The NESC problem aims at calculating likelihoods for named entities
that are proposed by the output of the NER model. We define the NESC
problem as follows:
Problem Given a text T and a sub-sequence of tokens S in T , what is the
probability of the S being a named entity in T .
For instance in our previous example “I love San Francisco” what is the
probability of the sequence containing the two tokens “San” and “Francisco”
being a named entity? In this example

T :
S:

I love San Francisco
{San, Francisco}

and S is the output of the NER model.

5

O-not-an-entity
B-Person
I-Person
B-Place
I-Place
B-Product
I-Product
B-Organization
I-Organization
B-Other
I-Other

I
0.985
0.001
0.001
0.002
0.001
0.003
0.001
0.001
0.001
0.001
0.003

love
San
0.988 0.012
0.001 0.023
0.001 0.001
0.001 0.483
0.001 0.006
0.001 0.276
0.001 0.159
0.002 0.009
0.001 0.011
0.001 0.012
0.002 0.008

Francisco
0.026
0.043
0.087
0.005
0.659
0.021
0.032
0.123
0.002
0.001
0.001

Table 1: Example of NER labels and their associated probabilities

3.2. NESC Model Architecture
By definition, NESC is a binary classification problem, and in the rest
of this section we discuss how we build a model for this problem. First we
define a context window around S with context size k. For instance if the
index of the starting token of S is i and the index of the ending token of S
is j, then the context window of S would be tokens i − k to j + k. Let’s
call this context window W . Needless to say we first pad the input text T
on both sides with pad size k to make sure that we can define W for any
sub-sequence of tokens in T . Figure 3.2 shows an example of context window
around a candidate named entity of size 2 tokens and context size (k) of 2.

Figure 3: NESC context window

6

Now NESC as a binary classification problem boils down to given a context window with context size k, whether the middle part of the window
(considering the context size k) is a named entity or not. To approach this
problem we need to vectorize the context window defined by the sequence of
the candidate tokens S and the other tokens in the context window. To do
this we use the output of pre-trained NER model’s bidirectional LSTM and
we build a sequence of the bidirectional LSTM layer output vectors that are
associated with the tokens of the context window of S. In other words the
vectorization of context window is simply the slice of the output of the pretrained NER model’s bidirectional LSTM layer output that coincides with
the context window.
Remember that for each token, the NER model’s bidirectional LSTM
vector associated with it has captured information from all other tokens that
come before and after it, and therefore the vectorization of context window
contains information from the entire text T and not only the context window
W.
Now that we have the context window W represented as a sequence of
vectors, the NESC problem can be viewed as a binary classification problem
on sequences. For this we build a model that is an LSTM from which we
take the internal state vector and follow that by a fully connected layer and
a softmax layer that outputs the probability of the candidate sub-sequence
S being a named entity. Figure 3.2 depicts the full architecture of the NESC
model.
3.3. Model Training
We build the training set for NESC from the labeled data of the NER
problem. Each NESC training sample is a sequence of vectors that are the
output of the bidirectional LSTM layer of the pre-trained NER model for a
context window, along with a target binary value which indicates whether
the center of the context window (with a given context size) is a named entity
or not. For positive samples (i.e., sub-sequences of tokens that are named
entities in the text that they appear in), we can directly use named entities
in the NER training data. Each labeled named entity in that data becomes
a record in the NESC training set with positive target value.
On the other hand, the negative samples are generated in two different
ways. First way of generating negative samples involves perturbations on
positive samples. More specifically, if we consider the window of tokens of
a positive sample in the text, we can get negative samples by extending,
7

Figure 4: NESC model architecture

shrinking, and moving this window. Table 2 shows some negative samples
that can be created by perturbing the correct named entity.
homeless
homeless
homeless
homeless
homeless

population
population
population
population
population

in San Francisco is surging
in San Francisco is surging
in San Francisco is surging
in San Francisco is surging
in San Francisco is surging

Positive NESC Sample
Negative NESC Sample
Negative NESC Sample
Negative NESC Sample
Negative NESC Sample

Table 2: Positive and negative samples for NESC from NER labeled data

As it is shown for each named entity in the NER labeled data, one positive
and several negative NESC samples can be created.
The second approach of generating negative samples for NESC is simply
taking a random sub-sequence of tokens from the NER labeled text. For
each random sub-sequence we also check to make sure that the sub-sequence
8

is not in fact an entity. This is easy to do since these random sub-sequences
are selected from NER labeled texts. For each random sub-sequence we first
sample the size of the sub-sequence from the empirical discrete distribution of
size (number of tokens) of the named entities in the NER labeled data. Next
we select a random sub-sequence of tokens of length that we just sampled.
Lastly we check to make sure that the selected random sub-sequence is not
a named entity. This can be automatically done by checking the labels of
tokens of the random sub-sequence.
Due to the imbalance in the number of positive and negative training
samples created, we use a weighted cross entropy loss function for NESC
in which the weights of the binary class are calculated based on number of
positive and negative samples in the training set.
3.4. NER Labeled Data
In order to create the labeled data for NER we first took a sample of
Tweets. Twitter’s Human Computations Team (HCOMP) labeled the tokens
of each Tweet sample based on the IOB schema. Each sample was initially
labeled by two different individuals and for samples that the two human
labels did not match, a third labeler also labeled the samples. At the end
we ended up with around 100,000 labeled English Tweets on which at least
2 labelers completely agreed on the labels. From this set we created our
training, validation, and test sets.
In our NER training set there are 62,507 named entities, each of which
would become a positive sample for NESC. On the other hand, using the
approaches discussed earlier we create 226,067 negative NESC samples, and
as a result our NESC training set contains 288,574 records.
4. Results
We first report the performance of our NER model on our labeled test
set and compare it with the performance of Stanford NLP’s [9] NER model.
Table 3 summarizes this performance comparison. Here we look at 3 performance measures:
• Untyped Token Level: The classification problem here is defined on
token labels (whether a token gets an entity or a not-an-entity label). A
token’s label would either associate it with an entity (label starts with
“B-” or “I-”) or associate it with not an entity (label O-not-an-entity).
9

In this case relevant instances are tokens that have labels other than
O-not-an-entity in the test set.
• Untyped Entity Level: The classification problem here is define on
named entities (whether an entity is correctly identified). Here named
entity types (Person, Place, etc.) are disregarded. In this case relevant
instance are full labeled named entities without their types in the test
set. Here the focus is on detecting full named entities and not on their
detected type.
• Typed Entity Level: The classification problem here is defined on
typed named entities (whether an entity and its type is correctly identified). In this case relevant indices are full labeled named entities and
their types in the test set. Here the focus is on detecting full named
entities along with their types.
Measure Type
Untyped Token Level
Untyped Entity Level
Typed Entity Level

Twitter NER
Stanford NLP NER
Twitter NER
Stanford NLP NER
Twitter NER
Stanford NLP NER

Precision
0.84
0.77
0.76
0.63
0.69
0.54

Recall
0.78
0.53
0.71
0.39
0.64
0.34

F1 Score
0.81
0.63
0.73
0.48
0.66
0.42

Table 3: Our NER model (Twitter NER) vs Stanford NLP NER model performance on
Tweets

As we can see from these numbers our NER model performs significantly
better than Stanford NLP’s NER model on Tweets with respect to precision, recall, and F1 score at both typed and untyped entity levels. This
performance difference is especially more apparent in recall numbers where
for instance the typed entity recall of our model is 0.64 whereas the same
value for Stanford NLP NER is 0.34.
Next we study the performance of our NESC model. We created training,
validation, and test sets for NESC by applying the method mentioned in
Section 3.3 on our NER’s training, validation, and test sets, respectively. We
calculate the precision and recall of the NESC model after isotonic calibration
on the validation set at different classification threshold values for the test
set. Figure 4 shows this Precision–Recall curve.
One could see from this curve that at 0.90 precision the recall is around
0.52 which means that more than half of the entities in the Tweets in the
10

Figure 5: NESC Precision–Recall curve

test set are detected with 0.90 precision. The sparsity in this precision–recall
curve is due to the small size of the test set (built using only 2000 Tweets)
and could be remedied by getting more labeled Tweets in order to get a
higher resolution precision–recall curve.
Table 4 shows the result of NER and NESC on a number of sample
Tweets. Each row in this table contains a tokenized Tweet text along with
named entities in the text detected by NER (highlighted in yellow), the entity
type detected by NER (shown as a subscript in red), and the probability of
the detected entity being in fact an entity found by NESC (also shown as a
subscript in red). For instance
“Barack Obama

person 0.993

is the 44th president of the United States”

indicates that NER has detected that “Barack Obama” is a person’s name
because NER’s label for “Barack” is B–Person, for “Obama” is I–Person, and
for “is” is O–not–and–entity. Moreover, according to NESC the probability
of the substring “Barack Obama” in the string “Barack Obama is the 44th
president of the United States” being an entity is 0.993.
In row 1 of the Table 4 we see that NER correctly has detected the song
11

1

2

3

4

5

6

7

8

9

10
11

12

Entities detected by NER with NESC probabilities in
tokenized Tweets
NowPlaying No Cigarette Smoking In My Room other 0.961 Stephen Marley person 0.973 ft Melonie Fiona person 0.985
https://t.co/mrlTZ0dmx4 16 : 39
@SInow : Tony Romo person 0.985 reportedly is unlikely to play
Sunday vs . the Steelers organization 0.978
https://t.co/WnWU8zaw4f https://t.co/f4hjMFMkoS
@GuardianBooks : The Essex Serpent other 0.912 beats Harry Potter
person 0.978 to win Waterstones organization 0.994 book of the year
https://t.co/hNEQJnRz0p
Marco Republic Paris Memory Foam Cushion Womens Mary
product 0.837 Jane Platform Wedges Heels Comfort Pumps
https://t.co/s1XZ19Oqt4 #pumps #flats #heels

Other candidates
probability

Marco Republic Paris
Memory Foam
Cushion Womens
Mary Jane Platform
Wedges Heels Comfort
Pumps 0.925

See How Bobrisky person 1.000 And Lolo person 1.000 Was Dancing
On Stage As Fans React [ Video ] https://t.co/PGUTaIztMT
https://t.co/Y5zv0Jnjrr
@90sNiallftafi : when calum person 1.000 and Michael person 1.000 got
Ashton person 1.000 to get a spider out of the bathroom & Ashton
scared calum https://t.co/5ieqdZAWTN
@DaiIyRap : Childish Gambino person 0.978 is dropping his new
album “ Awaken , My Love other 0.994 ” Next Month
https://t.co/qg5a42vnaa
Palmetto Packings Compression product 0.645 Seals - 5 / 16 ” Square
- FDA Listed - 20 Feet - New https://t.co/MmiPthCQNS
https://t.co/xsXX4GVO5i

Palmetto Packings
Compression Seals
0.837

FDA

0.943

First impressions : Russell Wilson person 0.985 and Seahawks
organization 1.000 sputter on offense in loss to Bucs organization 0.994
https://t.co/fTyvUduM17
Power Forward with us tonight at our President’s Community
Lecture other 0.925 featuring Bill Ritter person 0.973 .
https://t.co/YD1bOqp5bT https://t.co/x5tlAdPGEV
Happy Veterans day other 0.961 to every soldier who has fought for
the rights of our country @Justin martin3 love you
Long Island Volleyball organization 0.148 College Showcase at
SPORTIME organization 1.000 Thanks to all the Players and College
Coaches for making it a huge success ! #LIVCS
https://t.co/7KzLeH8D8a

Long Island 0.836
Long Island Volleyball
0.223

Long Island Volleyball
College Showcase0.285

Table 4: NER and NESC results on sample Tweets

name “No Cigarette Smoking In My Room” as entity of type other and NESC
gives the probability of 0.961 to this entity. Note that this is a rather long
name with 6 tokens which NER has correctly identified. Also there are 2
other named entities (persons) in the Tweet that are correctly identified by
NER and receive high NESC scores. Row 4 shows a Tweet in which a NER
does not identify a product name correctly and consequently the NESC score
12

that the incorrect subsequence gets is relatively lower than the NESC score
for the correct entity (last column). In row 5 we see a Tweet text in which all
the words are capitalized, which is quite common in Tweets. In this example
2 persons names are correctly identified by NER and both of them receive a
score of 1.0 from NESC. In row 6 we see a lower case named entity “calum”
is correctly identified by NER and receive very high scores from NESC. In
Row 8 we see that NER incorrectly returns the sequence “Palmetto Packings
Compression” as a product name where in fact the correct product name is
“Palmetto Packings Compression Seals”. However it is worth noting that the
NESC score for the the incorrect product name is quite low at 0.645, where as
if we query the correct product name from NESC the returned score is 0.837
(column 3). Moreover in this example NER fails to detect “FDA” as an entity,
but if we query NESC for FDA in that Tweet, the returned score is quite
high at 0.943. In row 11 we can see that NER correctly identifies “Veterans
day” as an entity of type other which has a high NESC score of 0.961. Finally
in row 12 we see that NER incorrectly identifies “Island Volleyball” as an
organization, but we see that the NESC score of that subsequence is very
low at 0.148.
For our application of recommending users named entities from Tweets,
from these results we see that we can rely on NESC scores with a high threshold. Specially if for a collection of Tweets NESC gives very high scores (say
all 0.95 or higher) to a given sequence of tokens, then we could confidently
assume that said subsequence of tokens in fact is an entity, and build user
recommendations based on that entity accordingly.
5. References
References
[1] D. Nadeau, S. Sekine, A survey of named entity recognition and classification, Lingvistic Investigationes 30 (2007) 3–26.
[2] C. N. dos Santos, V. Guimarães, Boosting named entity recognition
with neural character embeddings, CoRR abs/1505.05008 (2015).
[3] R. Socher, Y. Bengio, C. D. Manning, Deep learning for nlp (without
magic), in: Tutorial Abstracts of ACL 2012, ACL ’12, Association for
Computational Linguistics, Stroudsburg, PA, USA, 2012, pp. 5–5.

13

[4] A. Ritter, S. Clark, Mausam, O. Etzioni, Named entity recognition in
tweets: An experimental study, in: Proceedings of the Conference on
Empirical Methods in Natural Language Processing, EMNLP ’11, Association for Computational Linguistics, Stroudsburg, PA, USA, 2011,
pp. 1524–1534.
[5] X. Liu, S. Zhang, F. Wei, M. Zhou, Recognizing named entities in
tweets, in: Proceedings of the 49th Annual Meeting of the Association
for Computational Linguistics: Human Language Technologies - Volume
1, HLT ’11, Association for Computational Linguistics, Stroudsburg,
PA, USA, 2011, pp. 359–367.
[6] X. Liu, F. Wei, S. Zhang, M. Zhou, Named entity recognition for tweets,
ACM Trans. Intell. Syst. Technol. 4 (2013) 3:1–3:15.
[7] N. Limsopatham, N. Collier, Bidirectional lstm for named entity recognition in twitter messages, in: NUT@COLING.
[8] Z. Huang, W. Xu, K. Yu, Bidirectional LSTM-CRF models for sequence
tagging, CoRR abs/1508.01991 (2015).
[9] C. D. Manning, M. Surdeanu, J. Bauer, J. Finkel, S. J. Bethard, D. McClosky, The Stanford CoreNLP natural language processing toolkit, in:
Association for Computational Linguistics (ACL) System Demonstrations, pp. 55–60.
[10] IOB, Inside–outside–beginning (tagging) — Wikipedia,
the
free
encyclopedia,
2013.
https://en.wikipedia.org/wiki/
Inside-outside-beginning_(tagging)[Online; accessed 18-October2017].
[11] Q. Tran, A. MacKinlay, A. Jimeno-Yepes, Named entity recognition with stack residual LSTM and trainable bias decoding, CoRR
abs/1706.07598 (2017).
[12] J. P. C. Chiu, E. Nichols, Named entity recognition with bidirectional
lstm-cnns, CoRR abs/1511.08308 (2015).
[13] J. Pennington, R. Socher, C. D. Manning, Glove: Global vectors for
word representation, in: Empirical Methods in Natural Language Processing (EMNLP), pp. 1532–1543.
14

