Named Entity Recognition and Classification with Scikit-Learn | by Susan Li | Towards Data Science

Named Entity Recognition and Classification (NERC) is a process of recognizing information units like names, including person, organization and location names, and numeric expressions including time, date, money and percent expressions from unstructured text. The goal is to develop practical and domain-independent techniques in order to detect named entities with high accuracy automatically.
Last week, we gave an introduction on Named Entity Recognition (NER) in NLTK and SpaCy. Today, we go a step further, — training machine learning models for NER using some of Scikit-Learn’s libraries. Let’s get started!
The data is feature engineered corpus annotated with IOB and POS tags that can be found at Kaggle. We can have a quick peek of first several rows of the data.
Essential info about entities:
geo = Geographical Entity
org = Organization
per = Person
gpe = Geopolitical Entity
tim = Time indicator
art = Artifact
eve = Event
nat = Natural Phenomenon
Inside–outside–beginning (tagging)
The IOB (short for inside, outside, beginning) is a common tagging format for tagging tokens.
I- prefix before a tag indicates that the tag is inside a chunk.
B- prefix before a tag indicates that the tag is the beginning of a chunk.
An O tag indicates that a token belongs to no chunk (outside).
import pandas as pdimport numpy as npfrom sklearn.feature_extraction import DictVectorizerfrom sklearn.feature_extraction.text import HashingVectorizerfrom sklearn.linear_model import Perceptronfrom sklearn.model_selection import train_test_splitfrom sklearn.linear_model import SGDClassifierfrom sklearn.linear_model import PassiveAggressiveClassifierfrom sklearn.naive_bayes import MultinomialNBfrom sklearn.metrics import classification_report
The entire data set can not be fit into the memory of a single computer, so we select the first 100,000 records, and use Out-of-core learning algorithms to efficiently fetch and process the data.
df = pd.read_csv('ner_dataset.csv', encoding = "ISO-8859-1")df = df[:100000]df.head()
df.isnull().sum()
We notice that there are many NaN values in ‘Sentence #” column, and we fill NaN by preceding values.
df = df.fillna(method='ffill')df['Sentence #'].nunique(), df.Word.nunique(), df.Tag.nunique()
(4544, 10922, 17)
We have 4,544 sentences that contain 10,922 unique words and tagged by 17 tags.
The tags are not evenly distributed.
df.groupby('Tag').size().reset_index(name='counts')
The following code transform the text date to vector using DictVectorizer and then split to train and test sets.
X = df.drop('Tag', axis=1)v = DictVectorizer(sparse=False)X = v.fit_transform(X.to_dict('records'))y = df.Tag.valuesclasses = np.unique(y)classes = classes.tolist()X_train, X_test, y_train, y_test = train_test_split(X, y, test_size = 0.33, random_state=0)X_train.shape, y_train.shape
((67000, 15507), (67000,))
We will try some of the out-of-core algorithms that are designed to process data that is too large to fit into a single computer memory that support partial_fit method.
per = Perceptron(verbose=10, n_jobs=-1, max_iter=5)per.partial_fit(X_train, y_train, classes)
Because tag “O” (outside) is the most common tag and it will make our results look much better than they actual are. So we remove tag “O” when we evaluate classification metrics.
new_classes = classes.copy()new_classes.pop()new_classes
print(classification_report(y_pred=per.predict(X_test), y_true=y_test, labels=new_classes))
Linear classifiers with SGD training
sgd = SGDClassifier()sgd.partial_fit(X_train, y_train, classes)
print(classification_report(y_pred=sgd.predict(X_test), y_true=y_test, labels=new_classes))
Naive Bayes classifier for multinomial models
nb = MultinomialNB(alpha=0.01)nb.partial_fit(X_train, y_train, classes)
print(classification_report(y_pred=nb.predict(X_test), y_true=y_test, labels = new_classes))
Passive Aggressive Classifier
pa =PassiveAggressiveClassifier()pa.partial_fit(X_train, y_train, classes)
print(classification_report(y_pred=pa.predict(X_test), y_true=y_test, labels=new_classes))
None of the above classifiers produced satisfying results. It is obvious that it is not going to be easy to classify named entities using regular classifiers.
CRFs is often used for labeling or parsing of sequential data, such as natural language processing and CRFs find applications in POS Tagging, named entity recognition, among others.
We will train a CRF model for named entity recognition using sklearn-crfsuite on our data set.
import sklearn_crfsuitefrom sklearn_crfsuite import scorersfrom sklearn_crfsuite import metricsfrom collections import Counter
The following code is to retrieve sentences with their POS and tags. Thanks Tobias for the tip.
class SentenceGetter(object):        def __init__(self, data):        self.n_sent = 1        self.data = data        self.empty = False        agg_func = lambda s: [(w, p, t) for w, p, t in zip(s['Word'].values.tolist(),                                                            s['POS'].values.tolist(),                                                            s['Tag'].values.tolist())]        self.grouped = self.data.groupby('Sentence #').apply(agg_func)        self.sentences = [s for s in self.grouped]            def get_next(self):        try:             s = self.grouped['Sentence: {}'.format(self.n_sent)]            self.n_sent += 1            return s         except:            return Nonegetter = SentenceGetter(df)sentences = getter.sentences
Feature Extraction
Next, we extract more features (word parts, simplified POS tags, lower/title/upper flags, features of nearby words) and convert them to sklearn-crfsuite format — each sentence should be converted to a list of dicts. The following code were taken from sklearn-crfsuites official site.
def word2features(sent, i):    word = sent[i][0]    postag = sent[i][1]        features = {        'bias': 1.0,         'word.lower()': word.lower(),         'word[-3:]': word[-3:],        'word[-2:]': word[-2:],        'word.isupper()': word.isupper(),        'word.istitle()': word.istitle(),        'word.isdigit()': word.isdigit(),        'postag': postag,        'postag[:2]': postag[:2],    }    if i > 0:        word1 = sent[i-1][0]        postag1 = sent[i-1][1]        features.update({            '-1:word.lower()': word1.lower(),            '-1:word.istitle()': word1.istitle(),            '-1:word.isupper()': word1.isupper(),            '-1:postag': postag1,            '-1:postag[:2]': postag1[:2],        })    else:        features['BOS'] = True    if i < len(sent)-1:        word1 = sent[i+1][0]        postag1 = sent[i+1][1]        features.update({            '+1:word.lower()': word1.lower(),            '+1:word.istitle()': word1.istitle(),            '+1:word.isupper()': word1.isupper(),            '+1:postag': postag1,            '+1:postag[:2]': postag1[:2],        })    else:        features['EOS'] = Truereturn featuresdef sent2features(sent):    return [word2features(sent, i) for i in range(len(sent))]def sent2labels(sent):    return [label for token, postag, label in sent]def sent2tokens(sent):    return [token for token, postag, label in sent]
Split train and test sets
X = [sent2features(s) for s in sentences]y = [sent2labels(s) for s in sentences]X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.33, random_state=0)
Train a CRF model
crf = sklearn_crfsuite.CRF(    algorithm='lbfgs',    c1=0.1,    c2=0.1,    max_iterations=100,    all_possible_transitions=True)crf.fit(X_train, y_train)
Evaluation
y_pred = crf.predict(X_test)print(metrics.flat_classification_report(y_test, y_pred, labels = new_classes))
Way better! We will stick to sklearn-crfsuite and explore more!
What our classifier learned?
def print_transitions(trans_features):    for (label_from, label_to), weight in trans_features:        print("%-6s -> %-7s %0.6f" % (label_from, label_to, weight))print("Top likely transitions:")print_transitions(Counter(crf.transition_features_).most_common(20))print("\nTop unlikely transitions:")print_transitions(Counter(crf.transition_features_).most_common()[-20:])
Interpretation: It is very likely that the beginning of a geographical entity (B-geo) will be followed by a token inside geographical entity (I-geo), but transitions to inside of an organization name (I-org) from tokens with other labels are penalized hugely.
Check the state features
def print_state_features(state_features):    for (attr, label), weight in state_features:        print("%0.6f %-8s %s" % (weight, label, attr))print("Top positive:")print_state_features(Counter(crf.state_features_).most_common(30))print("\nTop negative:")print_state_features(Counter(crf.state_features_).most_common()[-30:])
Observations:
1). 5.183603 B-tim word[-3]:day The model learns that if a nearby word was “day” then the token is likely a part of a Time indicator.
2). 3.370614 B-per word.lower():president The model learns that token "president" is likely to be at the beginning of a person name.
3). -3.521244 O postag:NNP The model learns that proper nouns are often entities.
4). -3.087828 O word.isdigit() Digits are likely entities.
5). -3.233526 O word.istitle() TitleCased words are likely entities.
ELI5 is a Python package which allows to check weights of sklearn_crfsuite.CRF models.
Inspect model weights
import eli5eli5.show_weights(crf, top=10)
Observations:
It does make sense that I-entity must follow B-entity, such as I-geo follows B-geo, I-org follows B-org, I-per follows B-per, and so on.We can also see that it is not common in this data set to have a person right after an organization name (B-org -> I-per has a large negative weight).The model learned large negative weights for impossible transitions like O -> I-geo, O -> I-org and O -> I-tim, and so on.
It does make sense that I-entity must follow B-entity, such as I-geo follows B-geo, I-org follows B-org, I-per follows B-per, and so on.
We can also see that it is not common in this data set to have a person right after an organization name (B-org -> I-per has a large negative weight).
The model learned large negative weights for impossible transitions like O -> I-geo, O -> I-org and O -> I-tim, and so on.
For easy to read, we can check only a subset of tags.
eli5.show_weights(crf, top=10, targets=['O', 'B-org', 'I-per'])
Or check only some of the features for all tags.
eli5.show_weights(crf, top=10, feature_re='^word\.is',                  horizontal_layout=False, show=['targets'])
That was it, for now. I enjoyed making my hands dirty on sklearn-crfsuite and ELI5, hope you did too. Source code can be found at Github. Have a great week!