Named Entity Recognition with NLTK and SpaCy | by Susan Li | Towards Data Science

Named entity recognition (NER)is probably the first step towards information extraction that seeks to locate and classify named entities in text into pre-defined categories such as the names of persons, organizations, locations, expressions of times, quantities, monetary values, percentages, etc. NER is used in many fields in Natural Language Processing (NLP), and it can help answering many real-world questions, such as:
Which companies were mentioned in the news article?
Were specified products mentioned in complaints or reviews?
Does the tweet contain the name of a person? Does the tweet contain this person’s location?
This article describes how to build named entity recognizer with NLTK and SpaCy, to identify the names of things, such as persons, organizations, or locations in the raw text. Let’s get started!
import nltkfrom nltk.tokenize import word_tokenizefrom nltk.tag import pos_tag
I took a sentence from The New York Times, “European authorities fined Google a record $5.1 billion on Wednesday for abusing its power in the mobile phone market and ordered the company to alter its practices.”
ex = 'European authorities fined Google a record $5.1 billion on Wednesday for abusing its power in the mobile phone market and ordered the company to alter its practices'
Then we apply word tokenization and part-of-speech tagging to the sentence.
def preprocess(sent):    sent = nltk.word_tokenize(sent)    sent = nltk.pos_tag(sent)    return sent
Let’s see what we get:
sent = preprocess(ex)sent
We get a list of tuples containing the individual words in the sentence and their associated part-of-speech.
Now we’ll implement noun phrase chunking to identify named entities using a regular expression consisting of rules that indicate how sentences should be chunked.
Our chunk pattern consists of one rule, that a noun phrase, NP, should be formed whenever the chunker finds an optional determiner, DT, followed by any number of adjectives, JJ, and then a noun, NN.
pattern = 'NP: {<DT>?<JJ>*<NN>}'
Using this pattern, we create a chunk parser and test it on our sentence.
cp = nltk.RegexpParser(pattern)cs = cp.parse(sent)print(cs)
The output can be read as a tree or a hierarchy with S as the first level, denoting sentence. we can also display it graphically.
IOB tags have become the standard way to represent chunk structures in files, and we will also be using this format.
from nltk.chunk import conlltags2tree, tree2conlltagsfrom pprint import pprintiob_tagged = tree2conlltags(cs)pprint(iob_tagged)
In this representation, there is one token per line, each with its part-of-speech tag and its named entity tag. Based on this training corpus, we can construct a tagger that can be used to label new sentences; and use the nltk.chunk.conlltags2tree() function to convert the tag sequences into a chunk tree.
With the function nltk.ne_chunk(), we can recognize named entities using a classifier, the classifier adds category labels such as PERSON, ORGANIZATION, and GPE.
ne_tree = ne_chunk(pos_tag(word_tokenize(ex)))print(ne_tree)
Google is recognized as a person. It’s quite disappointing, don’t you think so?
SpaCy’s named entity recognition has been trained on the OntoNotes 5 corpus and it supports the following entity types:
import spacyfrom spacy import displacyfrom collections import Counterimport en_core_web_smnlp = en_core_web_sm.load()
We are using the same sentence, “European authorities fined Google a record $5.1 billion on Wednesday for abusing its power in the mobile phone market and ordered the company to alter its practices.”
One of the nice things about Spacy is that we only need to apply nlp once, the entire background pipeline will return the objects.
doc = nlp('European authorities fined Google a record $5.1 billion on Wednesday for abusing its power in the mobile phone market and ordered the company to alter its practices')pprint([(X.text, X.label_) for X in doc.ents])
European is NORD (nationalities or religious or political groups), Google is an organization, $5.1 billion is monetary value and Wednesday is a date object. They are all correct.
During the above example, we were working on entity level, in the following example, we are demonstrating token-level entity annotation using the BILUO tagging scheme to describe the entity boundaries.
pprint([(X, X.ent_iob_, X.ent_type_) for X in doc])
"B" means the token begins an entity, "I" means it is inside an entity, "O" means it is outside an entity, and "" means no entity tag is set.
Now let’s get serious with SpaCy and extracting named entities from a New York Times article, — “F.B.I. Agent Peter Strzok, Who Criticized Trump in Texts, Is Fired.”
from bs4 import BeautifulSoupimport requestsimport redef url_to_string(url):    res = requests.get(url)    html = res.text    soup = BeautifulSoup(html, 'html5lib')    for script in soup(["script", "style", 'aside']):        script.extract()    return " ".join(re.split(r'[\n\t]+', soup.get_text()))ny_bb = url_to_string('https://www.nytimes.com/2018/08/13/us/politics/peter-strzok-fired-fbi.html?hp&action=click&pgtype=Homepage&clickSource=story-heading&module=first-column-region&region=top-news&WT.nav=top-news')article = nlp(ny_bb)len(article.ents)
188
There are 188 entities in the article and they are represented as 10 unique labels:
labels = [x.label_ for x in article.ents]Counter(labels)
The following are three most frequent tokens.
items = [x.text for x in article.ents]Counter(items).most_common(3)
Let’s randomly select one sentence to learn more.
sentences = [x for x in article.sents]print(sentences[20])
Let’s run displacy.render to generate the raw markup.
displacy.render(nlp(str(sentences[20])), jupyter=True, style='ent')
One miss-classification here is F.B.I. It is hard, isn’t it?
Using spaCy’s built-in displaCy visualizer, here’s what the above sentence and its dependencies look like:
displacy.render(nlp(str(sentences[20])), style='dep', jupyter = True, options = {'distance': 120})
Next, we verbatim, extract part-of-speech and lemmatize this sentence.
[(x.orth_,x.pos_, x.lemma_) for x in [y                                       for y                                      in nlp(str(sentences[20]))                                       if not y.is_stop and y.pos_ != 'PUNCT']]
dict([(str(x), x.label_) for x in nlp(str(sentences[20])).ents])
Named entity extraction are correct except “F.B.I”.
print([(x, x.ent_iob_, x.ent_type_) for x in sentences[20]])
Finally, we visualize the entity of the entire article.
Try it yourself. It was fun! Source code can be found on Github. Happy Friday!