Sentiment analysis with BERT in PyTorch
By: Jakub Vonšovský
Tags: Machine Learning, Sentiment Analysis, NLP, Pytorch, Bert

That day in autumn of 2018 behind the walls of some Google lab has everything changed. The BERT was born.

Photo by Franck V. on Unsplash

With recent advances in the field of NLP, running such tasks as your own sentiment analysis is just a matter of minutes. I will show you how to build one, predicting whether movie reviews on IMDB are either positive or negative. So make a water for coffee. This won’t take more than one cup. Or two…

1. Building blocks

What is BERT

BERT stands for `Bidirectional Encoder Representation for Transformers` and provides pre-trained representation of language. Simply speaking, it converts any word or sentence to a list of vectors that points somewhere into space of all words and can be used for various tasks in potentially any given language. Of course, you need to have your BERT neural network trained on that language first, but usually someone else already did that for you from Wikipedia or BookCorpus dataset.

BERT is something like swiss army knife for NLP. Thanks to it, you don’t need to have theoretical background from computational linguistics and read dozens of books full of dust just to worsen your allergies. Its embedding space (fancy phrase for those vectors I mentioned above) can be used for sentiment analysis, named entity recognition, question answering, text summarization and others, while single-handedly outperforming almost all other existing models and sometimes even humans. Here I’ll demonstrate the first task mentioned.

What is Sentiment Analysis

No, it’s not about your memories of old house smell and how food was better in the past. Sentiment analysis deals with emotions in text. I’ll deal with simple binary positive / negative classification, but it can be fine-grained to neutral, strongly opinionated or even sad and happy.

The possibilities are countless. From getting back to angry users on your mobile app in the store to analyse what media think about bitcoins, so you can guess if the price will go up or down. This article will be about how to predict whether movie review on IMDB is negative or positive as this dataset is well known and publicly available.

PyTorch has arrived

If you are asking the eternal question “Why PyTorch and not Tensorflow as everywhere else?” I assume the answer “because this article already exists in Tensorflow” is not satisfactory enough. So I will give you a better one.

Back in the old days of summer 2019 when we were digging out potentially useful NLP projects from repos at my job, it was using Tensorflow. Widely used framework from Google that helped to bring deep learning to masses. But nowadays, 1.x seems quite outdated. The way how you have to build graphs before using them, raises eyebrows. You have to build a computational graph even for saving your precious model. And there are bugs. It recomputes the whole graph every time you are predicting from already existing model, eating precious time of your customer in the production mode. But no worries, you can hack this bug by saving your model and reloading it. Just in different way than normally saving model for later use. Wait… what? There are two ways of saving weights? And then there are versioning problems…

PyTorch is more straightforward. You just imperatively stack layer after layer of your neural network with one liners. If you ever used Numpy then good for you. PyTorch is like Numpy for deep learning. And you save your models with one liners. No extra code required. Also “everywhere else” is no longer valid at least in academic world, where PyTorch has already taken over Tensorflow in usage. Meet the new King of deep learning realm.

There is great implementation of BERT in PyTorch called Transformers from HuggingFace. Apart from BERT, it contains also other models like smaller and faster DistilBERT or scary-dangerous-world-destroying GPT-2. And replacing Tensorflow based BERT in our project without affecting functionality or accuracy took less than week. Much less than we spent with solving seemingly endless TF issues. Before continuing reading this article, just install it with pip. It won’t hurt, I promise.

2. Building Sentiment Classifier

Great, we have basic building blocks — Pytorch and Transformers. From now on, it will be ride. Please download complete code described here from my GitHub.

Prepare BERT input

We need to read and preprocess IMDB reviews data. I chose simple format of one comment per line, where first 12500 lines are positive and the other half is negative. You can get this file from my Google Drive (along with pre-trained weights, more on that later on). If you are good with defaults, just locate script.py, create and put it into data/ folder.

Original source file is this IMDB dataset hosted on Stanford if you are interested in where it comes from. I just gave it some nicer format. However, there is still some work to do. Before passing to tokenizer, I removed some html characters that appear in those comments and since BERT uncased model is being used, also lowered characters.

def _parse_imdb_line(line):

line = line.strip().lower()

line = line.replace(" ", " ")

line = re.sub(r'<br(\s\/)?>', ' ', line)

line = re.sub(r' +', ' ', line) # merge multiple spaces to one return line

The next step is to convert words to numbers. You cannot just pass letters to neural networks. Apart from computer resources, it eats only numbers. So here comes BERT tokenizer. It splits entire sentence into list of tokens which are then converted into numbers. For example, “It was simply breathtaking.” is cut into [‘it’, ‘was’, ‘simply’, ‘breath’, ‘##taking’, ‘.’] and then mapped to [2009, 2001, 3432, 3052, 17904, 1012] according to their positions in vocabulary. Notice that some words are split into more tokens, to have less difficulties finding it in vocabulary.

BERT is also using special tokens CLS and SEP (mapped to ids 101 and 102) standing for beginning and end of a sentence. Because all such sentences have to have the same length, such as 256, the rest is padded with zeros. Wrapped everything together, our example will be fed into neural network as [101, 6919, 3185, 2440, 1997, 6569, 1012, 102, 0 * 248].

def convert_to_embedding(self, sentence):

tokens = self.tokenizer.tokenize(sentence)

tokens = tokens[:MAX_SENTENCE_LENGTH - 2]

bert_sent = self._rpad(self.tokenizer.convert_tokens_to_ids(["CLS"] + tokens + ["SEP"]), n=MAX_SENTENCE_LENGTH)



return bert_sent

Training

Now the computationally intensive part. While BERT model itself was already trained on language corpus by someone else and you don’t have to do anything by yourself, your duty is to train its sentiment classifier. Like telling your robot with fully functioning brain what is good and what is bad.

There’s not much to describe here. PyTorch training is somehow standardized and well described in many articles here on Medium. My model.py used for training / evaluation / prediction is just modified example file from Transformers repository. The only extra work done here is setting smaller learning rate for basic model as it is already well trained and bigger for classifier:

optimizer_grouped_parameters = [

{"params": model.bert.parameters(), "lr": 1e-5},

{"params": model.classifier.parameters(), "lr": 1e-3}

]

I also left behind some other hyperparameters for tuning such as `warmup steps` or `gradient accumulation steps` if anyone is interested to play with them. But describing them is beyond the scope of one cup of coffee time.

You should have downloaded dataset in data/ directory before running training. Transformers will take care of the rest automatically. It will download BERT model, vocab and config file into cache and will copy these files into output directory once the training is finished. Have a look at these later.

You can run training in your secret home lab equipped with GPU units as python script.py --train , put python notebook from notebooks/ directory into Google Colab GPU environment (it takes around 1 hour of training there) or just don’t do it and download already trained weights from my Google Drive. Default setting is to read them from weights/ directory for evaluation / prediction.

Prediction

Here comes that important part. The one that you can put into your API and use it for analyzing whether bitcoins go up or readers of your blog are mostly nasty creatures. It’s pretty straightforward. You need to convert your text into numbers as described above and then call first model.eval() and model(numbers) . See code for full reference. Run the script simply with:

python script.py --predict “That movie was so awful that I wanted to spill coke on everyone around me.”

3. Results

The interesting part telling you how much badass BERT is. Outperforming the others just with few lines of code. And 440 MB of neural network weights.

precision recall f1-score support

0.0 0.92 0.92 0.92 12500

1.0 0.92 0.92 0.92 12500

accuracy 0.92 25000

macro avg 0.92 0.92 0.92 25000

weighted avg 0.92 0.92 0.92 25000

Whoa, 92 percent of accuracy! With almost no hyperparameter tuning. That is something. Have a look for example here :-P. Notice those nltk imports and all the sand picking around. This is how it was done in the old days.

Epilogue

Training sentiment classifier on IMDB reviews is one of benchmarks being used out there. This article was about showing you how powerful tools of deep learning can be. And how easy is to try them by yourself, because someone smart has already done the hard part for you. Now, with your own model that you can bend to your needs, you can start to explore what else BERT offers.

BERT is mighty. And this is not the end. Scientists around the globe work on better models that are even more accurate or using less parameters, such as DistilBERT, AlBERT or entirely new types built upon knowledge gained from BERT. Such as BERT was built on works like ELMO. The revolution has just started…

BTW if you don’t like reading articles and are rather jump-straight-to-the-end person, I am reminding the code link here. You can start to play with it right now.