A Hands-On Guide To Text Classification With Transformer Models (XLNet, BERT, XLM, RoBERTa) | by Thilina Rajapakse | Towards Data Science

Please consider using the Simple Transformers library as it is easy to use, feature-packed, and regularly updated. The article still stands as a reference to BERT models and is likely to be helpful with understanding how BERT works. However, Simple Transformers offers a lot more features, much more straightforward tuning options, all the while being quick and easy to use! The links below should help you get started quickly.
Binary ClassificationMulti-Class ClassificationMulti-Label ClassificationNamed Entity Recognition (Part-of-Speech Tagging)Question AnsweringSentence-Pair Tasks and RegressionConversational AILanguage Model Fine-TuningELECTRA and Language Model Training from ScratchVisualising Model Training
Binary Classification
Multi-Class Classification
Multi-Label Classification
Named Entity Recognition (Part-of-Speech Tagging)
Question Answering
Sentence-Pair Tasks and Regression
Conversational AI
Language Model Fine-Tuning
ELECTRA and Language Model Training from Scratch
Visualising Model Training
The Pytorch-Transformers (now Transformers) library has moved on quite a bit since this article was written. I recommend using SimpleTransformers as it is kept up to date with the Transformers library and is significantly more user-friendly. While the ideas and concepts in this article still stand, the code and the Github repo are no longer actively maintained.
Language Model Fine-TuningELECTRA and Language Model Training from ScratchVisualising Model Training
Language Model Fine-Tuning
ELECTRA and Language Model Training from Scratch
Visualising Model Training
I highly recommend cloning the Github repo for this article and running the code while you follow the guide. It should help you understand both the guide and the code better. Reading is great, but coding is better. 😉
Special thanks to Hugging Face for their Pytorch-Transformers library for making Transformer Models easy and fun to play with!
Transformer models have taken the world of Natural Language Processing by storm, transforming (sorry!) the field by leaps and bounds. New, bigger, and better models seem to crop up almost every month, setting new benchmarks in performance across a wide variety of tasks.
This post is intended as a straightforward guide to utilizing these awesome models for text classification tasks. As such, I won’t be talking about the theory behind the networks, or how they work under the hood. If you are interested in diving into the nitty-gritty of Transformers, my recommendation is Jay Alammar’s Illustrated Guides here.
This also serves as an update to my earlier guide on Using BERT for Binary Text Classification. I’ll be using the same dataset (Yelp Reviews) that I used the last time to avoid having to download a new dataset because I’m lazy and I have terrible internet. The motivation behind the update is down to several reasons, including the update to the HuggingFace library I used for the previous guide, as well as the release of multiple new Transformer models which have managed to knock BERT off its perch.
With the background set, let’s take a look at what we’ll be doing.
Setting up the development environment, with the Pytorch-Transformers library by HuggingFace.Converting .csv datasets to .tsv format used by the HuggingFace library.Setting up pre-trained models.Converting data into features.Fine-tuning models.Evaluation.
Setting up the development environment, with the Pytorch-Transformers library by HuggingFace.
Converting .csv datasets to .tsv format used by the HuggingFace library.
Setting up pre-trained models.
Converting data into features.
Fine-tuning models.
Evaluation.
I’ll be using two Jupyter Notebooks, one for data preparation, and one for training and evaluation.
It’s highly recommended to use a virtual environment when installing and working with various Python libraries. My personal favourite is Anaconda, but you can use anything you wish.conda create -n transformers python pytorch pandas tqdm jupyterconda activate transformersconda install -c anaconda scikit-learnpip install pytorch-transformerspip install tensorboardXPlease note that there may be additional packages used in the guide that are not installed here. If run into missing packages, simply install them via conda or pip.Linux users can use the shell script here to download and extract the Yelp Reviews Polarity dataset. Others can manually download it here at fast.ai. Also, direct download link.I placed the train.csv and test.csv files in a directory named data. <starting_directory>/data/
It’s highly recommended to use a virtual environment when installing and working with various Python libraries. My personal favourite is Anaconda, but you can use anything you wish.conda create -n transformers python pytorch pandas tqdm jupyterconda activate transformersconda install -c anaconda scikit-learnpip install pytorch-transformerspip install tensorboardXPlease note that there may be additional packages used in the guide that are not installed here. If run into missing packages, simply install them via conda or pip.
Linux users can use the shell script here to download and extract the Yelp Reviews Polarity dataset. Others can manually download it here at fast.ai. Also, direct download link.I placed the train.csv and test.csv files in a directory named data. <starting_directory>/data/
Most online datasets will typically be in .csv format. Following the norm, the Yelp dataset contains two csv files train.csv and test.csv.
Kicking off our first (data preparation) notebook, let’s load the csv files in with Pandas.
import pandas as pd
from sklearn.model_selection import train_test_split
from tqdm import tqdm_notebook

prefix = 'data/'

train_df = pd.read_csv(prefix + 'train.csv', header=None)
train_df.head()

test_df = pd.read_csv(prefix + 'test.csv', header=None)
test_df.head()

However, the labels used here break the norm by being 1 and 2 instead of the usual 0 and 1. I’m all for a bit of rebellion, but this just puts me off. Let’s fix this so that the labels are 0 and 1, indicating a bad review and good review respectively.
train_df[0] = (train_df[0] == 2).astype(int)
test_df[0] = (test_df[0] == 2).astype(int)

train_df.head()

test_df.head()

 

We need to do some final bit of retouching before our data is ready for the Pytorch-Transformer models. The data needs to be in tsv format, with four columns, and no header.
guid: An ID for the row.
label: The label for the row (should be an int).
alpha: A column of the same letter for all rows. Not used in classification but still needed.
text: The text for the row.
So, let’s get the data in order, and save it in tsv format.
train_df = pd.DataFrame({
    'id':range(len(train_df)),
    'label':train_df[0],
    'alpha':['a']*train_df.shape[0],
    'text': train_df[1].replace(r'\n', ' ', regex=True)
})

train_df.head()

dev_df = pd.DataFrame({
    'id':range(len(test_df)),
    'label':test_df[0],
    'alpha':['a']*test_df.shape[0],
    'text': test_df[1].replace(r'\n', ' ', regex=True)
})

dev_df.head()

train_df.to_csv('data/train.tsv', sep='\t', index=False, header=False)
dev_df.to_csv('data/dev.tsv', sep='\t', index=False, header=False)

 

This marks the end of the data preparation Notebook, and we’ll continue with the training Notebook from the next section.
Before we can start the actual training, we need to convert our data from text into numerical values that can be fed into neural networks. In the case of Transformer models, the data will be represented as InputFeature objects.
To make our data Transformer-ready, we’ll be using the classes and functions in the file utils.py. (Brace yourself, a wall of code incoming!)
Let’s look at the important bits.
The InputExample class represents a single sample of our dataset;
guid: a unique ID
text_a: Our actual text
text_b: Not used in classification
label: The label of the sample
The DataProcessor and BinaryProcessor classes are used to read in the data from tsv files and convert it into InputExamples.
The InputFeature class represents the pure, numerical data that can be fed to a Transformer.
The three functions convert_example_to_feature, convert_examples_to_features, _truncate_seq_pair are used to convert InputExamples into InputFeatures which will finally be sent to the Transformer model.
The conversion process includes tokenization, and converting all sentences to a given sequence length (truncating longer sequences, and padding shorter sequences). During tokenization, each word in the sentence is broken apart into smaller and smaller tokens (word pieces) until all the tokens in the dataset are recognized by the Transformer.
As a contrived example, let’s say we have the word understanding. The Transformer we are using does not have a token for understanding but it has separate tokens for understand and ing. Then, the word understanding would be broken into the tokens understand and ing. The sequence length is the number of such tokens in the sequence.
The convert_example_to_feature function takes a single sample of data and converts it into an InputFeature. The convert_examples_to_features function takes a list of examples and returns a list of InputFeatures by using the convert_example_to_feature function. The reason behind there being two separate functions is to allow us to use Multiprocessing in the conversion process. By default, I’ve set the process count to cpu_count() - 2, but you can change it by passing a value for the process_count parameter in the convert_examples_to_features function.
Now, we can go to our training notebook and import the stuff we’ll use and configure our training options.
Go through the args dictionary carefully and note all the different settings you can configure for training. In my case, I am using fp16 training to lower memory usage and speed up training. If you don’t have Nvidia Apex installed, you will have to turn off fp16 by setting it to False.
In this guide, I am using the XL-Net model with a sequence length of 128. Please refer to the Github repo for the full list of available models.
Now, we are ready to load our model for training.
The coolest thing about the Pytorch-Transformers library is that you can use any of the MODEL_CLASSES above, just by changing the model_type and model_name in the arguments dictionary. The process for fine-tuning, and evaluating is basically the same for all the models. All hail HuggingFace!
Next, we have functions defining how to load data, train a model, and to evaluate a model.
Finally, we have everything ready to tokenize our data and train our model.
It should be fairly straightforward from here.
This will convert the data into features and start the training process. The converted features will be automatically cached, and you can reuse them later if you want to run the same experiment. However, if you change something like the max_seq_length, you will need to reprocess the data. Same goes for changing the model used. To reprocess the data, simply set reprocess_input_data to True in the args dictionary.
For comparison, this dataset took about 3 hours for training on my RTX 2080.
Once training completes, we can save everything.
Evaluation is quite easy as well.
Without any parameter tuning, and with one training epoch, my results are as follows.
INFO:__main__:***** Eval results  *****INFO:__main__:  fn = 1238INFO:__main__:  fp = 809INFO:__main__:  mcc = 0.8924906867291726INFO:__main__:  tn = 18191INFO:__main__:  tp = 17762
Not too shabby!
Transformer models have displayed incredible prowess in handling a wide variety of Natural Language Processing tasks. Here, we’ve looked at how we can use them for one of the most common tasks, which is Sequence Classification.
The Pytorch-Transformers library by HuggingFace makes it almost trivial to harness the power of these mammoth models!
When working with your own datasets, I recommend editing the data_prep.ipynb notebook to save your data files as tsv files. For most cases, you should be able to get things running by simply making sure that the correct columns containing the text and the labels are passed to the train_df and the dev_df constructors. You could also define your own class that inherits from the DataProcessor class in the utils.py file, but I feel the first approach is simpler.
Please do use the Github repo as opposed to copying and pasting from the post here. Any fixes or extra features will be added to the Github repo and is unlikely to be added here unless it is a breaking change. Code is embedded to Medium using Gists, and as such, they are not automatically synced with the repo code.
If you need support, or you spot a bug, opening an issue on the Github repo will probably get a quicker response than comments on this article. It’s easy to miss comments on here, and the lack of comment/chat threads makes it difficult to follow. As a bonus, other people struggling with the same issue will probably be able to find the answer easier if it was on Github rather than on a Medium response.