Simple Transformers — Named Entity Recognition with Transformer Models | by Thilina Rajapakse | Towards Data Science

The Simple Transformers library was conceived to make Transformer models easy to use. Transformers are incredibly powerful (not to mention huge) deep learning models which have been hugely successful at tackling a wide variety of Natural Language Processing tasks. Simple Transformers enabled the application of Transformer models to Sequence Classification tasks (binary classification initially, but with multiclass classification added soon after) with only three lines of code.
I am delighted to announce that Simple Transformers now supports Named Entity Recognition, another common NLP task, alongside Sequence Classification.
Links to other capabilities:
Binary Sequence Classification with Simple Transformers
Multiclass Sequence Classification with Simple Transformers
The Simple Transformers library is built on top of the excellent Transformers library by Hugging Face. The Hugging Face Transformers library is the library for researchers and other people who need extensive control over how things are done. It is also the best choice when you need to stray off the beaten path, do things differently, or do new things altogether. Simple Transformers is, well, a lot simpler.
You want to try out that brilliant idea, you want to roll up your sleeves and get to work but the thousands of lines of code full of cryptic (but cool) looking stuff can be intimidating even to a veteran NLP researcher. The core idea behind Simple Transformers is that using Transformers doesn’t need to be difficult (or frustrating).
Simple Transformers abstracts away all the complicated setup code while retaining flexibility and room for configuration as far as possible. A Transformer model can be used in just three lines of code, one line for initializing, one for training, and one for evaluation.
This post demonstrates how to perform NER using Simple Transformers.
All source code is available on the Github Repo. If you have any issues or questions, that’s the place to resolve them. Please do check it out!
Install Anaconda or Miniconda Package Manager from hereCreate a new virtual environment and install the required packages.conda create -n transformers python pandas tqdmconda activate transformersIf using cuda:conda install pytorch cudatoolkit=10.0 -c pytorchelse:conda install pytorch cpuonly -c pytorchconda install -c anaconda scipyconda install -c anaconda scikit-learnpip install transformerspip install tensorboardx pip install seqevalInstall simpletransformers.pip install simpletransformers
Install Anaconda or Miniconda Package Manager from here
Create a new virtual environment and install the required packages.conda create -n transformers python pandas tqdmconda activate transformersIf using cuda:conda install pytorch cudatoolkit=10.0 -c pytorchelse:conda install pytorch cpuonly -c pytorchconda install -c anaconda scipyconda install -c anaconda scikit-learnpip install transformerspip install tensorboardx pip install seqeval
Install simpletransformers.pip install simpletransformers
To demonstrate Named Entity Recognition, we’ll be using the CoNLL Dataset. Getting hold of this dataset can be a little tricky, but I found a version of it on Kaggle that works for our purpose.
Download the dataset from Kaggle.Extract the text files to the data/ directory. (It should contain 3 text files train.txt, valid.txt, test.txt. We’ll be using the train and test files. You can use the valid file to perform hyperparameter tuning to improve model performance.
Download the dataset from Kaggle.
Extract the text files to the data/ directory. (It should contain 3 text files train.txt, valid.txt, test.txt. We’ll be using the train and test files. You can use the valid file to perform hyperparameter tuning to improve model performance.
Simple Transformers’ NER model can be used with either .txt files or with pandas DataFrames. For a usage example with DataFrames, please refer to the minimal start example for NER in the repo docs.
When using your own datasets, the input text files should follow the CoNLL format. Each line in the file should contain one word and its related tags separated by a single space each. Simple Transformers assumes the first “word” in a line is the actual word, and that the last “word” in a line is its assigned label. To denote a new sentence, an empty line is added between the last word of the previous sentence and the first word of the next sentence. However, it may be easier to use the DataFrame approach when using custom datasets.
We create a NERModel that can be used for training, evaluation, and prediction in NER tasks. The full parameter list for a NERModel object is given below.
model_type: The type of model (bert, roberta)
model_name: Default Transformer model name or path to a directory containing Transformer model file (pytorch_nodel.bin).
labels (optional): A list of all Named Entity labels. If not given, [“O”, “B-MISC”, “I-MISC”, “B-PER”, “I-PER”, “B-ORG”, “I-ORG”, “B-LOC”, “I-LOC”] will be used.
args (optional): Default args will be used if this parameter is not provided. If provided, it should be a dict containing the args that should be changed in the default args.
use_cuda (optional): Use GPU if available. Setting to False will force model to use CPU only.
To load a model a previously saved model instead of a default model, you can change the model_name to the path to a directory which contains a saved model.
model = NERModel(‘bert’, ‘path_to_model/’)
A NERModel contains a python dict args with many attributes that provide control over hyperparameters. A detailed description of each is provided in the repo docs. The default values are shown below.
Any of these attributes can be modified when creating a NERModel or when calling its train_model method by simply passing in a dict containing the key-value pairs to be updated. An example is given below.
As promised, training can be done in a single line of code.
The train_model method will create a checkpoint (save) of the model at every nth step where n is self.args['save_steps']. Upon completion of training, the final model will be saved to self.args['output_dir'].
Loading a saved model is shown below.
Again, evaluation is just a single line of code.
Here, the three return values are:
result: Dictionary containing evaluation results. (eval_loss, precision, recall, f1_score)
model_outputs: List of raw model outputs
preds_list: List of predicted tags
The evaluation results I obtained are given here for reference.
{'eval_loss': 0.10684790916955669, 'precision': 0.9023580786026201, 'recall': 0.9153082919914954, 'f1_score': 0.9087870525112148}
Not too shabby for a single run with default hyperparameter values!
In real-world applications, we often have no idea what the true labels should be. To perform predictions on arbitrary examples, you can use the predict method. This method is fairly similar to the eval_model method except that this takes in a list of text and returns a list of predictions and a list of model outputs.
predictions, raw_outputs = model.predict(["Some arbitary sentence"])
Simple Transformers provides a quick and easy way to perform Named Entity Recognition (and other token level classification tasks). To steal a line from the man behind BERT himself, Simple Transformers is “conceptually simple and empirically powerful”.