Named-Entity Recognition using Keras Bi-LSTM | Towards Data Science

Named-entity recognition (NER) (also known as entity identification, entity chunking and entity extraction) is a subtask of information extraction that seeks to locate and classify named entities mentioned in unstructured text into pre-defined categories such as person names, organizations, locations, medical codes, time expressions, quantities, monetary values, percentages, etc.
In this project, we will work with a NER dataset provided by Kaggle. The dataset can be accessed here. This dataset is the extract from the GMB corpus, which is tagged, annotated, and built specifically to train the classifier to predict named entities such as name, location, etc. Dataset also includes one additional feature (POS) that can be used in classification. In this project, however, we are working only with one feature sentence.
Let us begin by loading and visualizing the dataset. To download ner_dataset.csv, go to this link in Kaggle.
We will have to use encoding = ‘unicode_escape’ while loading the data. This function takes a parameter to toggle the wrapping quotes’ addition and escaping that quote’s quote in a string.
import pandas as pddata = pd.read_csv('ner_dataset.csv', encoding= 'unicode_escape')data.head()
Note that the sentences in the dataset are tokenized in the column “Word”. The column “sentence #” displays the sentence number once and then prints NaN until the next sentence begins. The ‘Tag’ column will be our label (y).
To train a neural network, we will use two mappings as given below.
{token} to {token id}: address the row in embeddings matrix for the current token.
{tag} to {tag id}: one-hot ground truth probability distribution vectors for computing the loss at the network’s output.
This step is desired in any machine learning model, including the neural-network, which requires integers as input.
data['Word_idx'] = data['Word'].map(token2idx)data['Tag_idx'] = data['Tag'].map(tag2idx)data.head()
We can see that the function has added two new index columns for our X (Word_idx) and y (Tag_idx) variables. Next, let us collect tokens into arrays in the respective sequence to make the recurrent neural network’s best use.
To transform columns into sequential arrays, we will
Fill NaN in the ‘sentence #’ column using method ffill in fillna.
After that, run groupby on the sentence column to get a list of tokens and tags.
# Fill nadata_fillna = data.fillna(method='ffill', axis=0)# Groupby and collect columnsdata_group = data_fillna.groupby(['Sentence #'],as_index=False)['Word', 'POS', 'Tag', 'Word_idx', 'Tag_idx'].agg(lambda x: list(x))# Visualise datadata_group.head()
Padding: The LSTM layers accept sequences of the same length only. Therefore, every sentence represented as integers (‘Word_idx’) must be padded to have the same length. We will work with the max length of the longest sequence and pad the shorter sequences to achieve this.
Please note we can use shorter padding lengths as well. In that case, you will be padding the shorter sequences and truncating the longer sequences.
We will also be converting the y variable as one-hot encoded vector using the to_categorical function in Keras. Let us import the required packages.
from sklearn.model_selection import train_test_splitfrom keras.preprocessing.sequence import pad_sequencesfrom keras.utils import to_categorical
Neural network models work with graphical structure. Therefore we will first need to design the architecture and set input and out dimensions for every layer. RNNs are capable of handling different input and output combinations. We will use many to many architectures for this ask. Refer to the last architecture in the image given below. Our task is to output tag (y) for a token (X) ingested at each time step.
Lets begin by loading, the required packages.
import numpy as npimport tensorflowfrom tensorflow.keras import Sequential, Model, Inputfrom tensorflow.keras.layers import LSTM, Embedding, Dense, TimeDistributed, Dropout, Bidirectionalfrom tensorflow.keras.utils import plot_model
Its always best to set seed for reproducibility.
from numpy.random import seedseed(1)tensorflow.random.set_seed(2)
To build a better understanding of the layers’ input and output dimensions, compare the layers in the model plot with the brief on each layer given in the image (model-plot) below.
In this architecture, we are primarily working with three layers (embedding, bi-lstm, lstm layers) and the 4th layer, which is TimeDistributed Dense layer, to output the result. We will discuss the layers in detail in the below sections.
Layer 1 — Embedding layer: We will specify the maximum length (104) of the padded sequences. After the network is trained, the embedding layer will transform each token into a vector of n dimensions. We have chosen the n dimensions to be (64).
The dimensions (?, 104, 64) that we see in the given neural network plot’s embedding layer comes from these specifications. Please note the 1st dimension shows ‘?’ or None in the plots. They represent the batch sizes. If it is not provided, they show ‘?’ or None, which means the model can take any batch size.
Layer 2 — Bidirectional LSTM: Bidirectional lstm takes a recurrent layer (e.g. the first LSTM layer) as an argument. This layer takes the output from the previous embedding layer (104, 64).
Since this is bidirectional lstm, we will have forward and backward outputs. Combine these outputs before passing ti to the next layer by summing or taking average or concatenating or multiplying. Find these functions in the merge mode argument in bi-lstm layer. The outputs are concatenated in the default mode, which in turn doubles the number of outputs to the next layer. In our case, it becomes 128(64 * 2).
Layer 3 — LSTM Layer: An LSTMnetwork is a recurrent neural network that has LSTM cell blocks in place of our standard neural network layers. These cells have various components called the input gate, forget gate, and output gate.
This layer takes the output dimension from the previous bidirectional lstm layer (?, 104, 128) and outputs (?, 104, 256)
Layer 4 — TimeDistributed Layer: We are dealing with Many to Many RNN Architecture, where we expect output from every input sequence. Here is an example, in the sequence (a1 →b1, a2 →b2...an →bn), a, and b are inputs and outputs of every sequence. The TimeDistributeDense layers allow Dense(fully-connected) operation across every output over every time-step. No using this layer will result in one final output.
This layer takes the output dimension of the previous lstm layer (104, 256) and outputs the max sequence length (104) and max tags (17).
input_dim = len(list(set(data['Word'].to_list())))+1output_dim = 64input_length = max([len(s) for s in data_group['Word_idx'].tolist()])n_tags = len(tag2idx)print('input_dim: ', input_dim, '\noutput_dim: ', output_dim, '\ninput_length: ', input_length, '\nn_tags: ', n_tags)
Given below is the code used to build the model architecture we discussed above. Before fitting the model, check the model graph by plotting the model using plot_model function or run model.summary() for the model summary.
We will fit the model with a for loop to save and visualize the loss at every epoch.
results = pd.DataFrame()model_bilstm_lstm = get_bilstm_lstm_model()plot_model(model_bilstm_lstm)results['with_add_lstm'] = train_model(train_tokens, np.array(train_tags), model_bilstm_lstm)
The model started with 0.9169 accuracy ended. After running 25 epochs with 1000 batch size, the final accuracy was 0.9687.
Please experiment with the model with different batch sizes, dropout value, optimizers, metrics, and layers to a better result. Find the complete code for this project in the Github Repository.
For anyone interested, here is a piece of code that will highlight the entities in any given sentence using spaCy.
import spacyfrom spacy import displacynlp = spacy.load('en_core_web_sm')text = nlp('Jim bought 300 shares of Acme Corp. in 2006. And producing an annotated block of text that highlights the names of entities: [Jim]Person bought 300 shares of [Acme Corp.]Organization in [2006]Time. In this example, a person name consisting of one token, a two-token company name and a temporal expression have been detected and classified.State-of-the-art NER systems for English produce near-human performance. For example, the best system entering MUC-7 scored 93.39% of F-measure while human annotators scored 97.60% and 96.95%.[1][2]')displacy.render(text, style = 'ent', jupyter=True)