Evolution of NLP — Part 4 — Transformers — BERT, XLNet, RoBERTa
By: Kanishk Jain
Tags: NLP, Deep Learning, AI, Machine Learning

Evolution of NLP — Part 4 — Transformers — BERT, XLNet, RoBERTa

Using SOTA Transformers models for Sentiment Classification

This is endgame! Transformers are one of the premier deep learning architectures used today, in combination with transfer learning to handle various NLP tasks. We continue our journey to find the best solution for our Sentiment Classification task. We’ve already seen basic techniques — TF-IDF and Bag of Words, then we saw LSTMs, next we applied Transfer Learning along with LSTMs and now we explore an altogether different architecture here!

Before we move ahead, it’s important to get a sense of issues with RNNs/LSTMs.

Issues with LSTMs

LSTMs have been very critical in the evolution of NLP. These were some of the most pivotal architectures that resolved issues within RNNs and made deep learning more wide-spread. But they pose some issues which make them difficult to work with.

Slow to train! RNNs are already slower to train because we need to provide data sequentially while training, instead of parallelly. A hidden state would require inputs from all the previous words to make any progress. This kind of architecture doesn’t take advantage of today’s GPUs which are optimized for parallel processing. Couple this with the added complexity in an LSTM, in the form of multiple gates, they are even slower to train. Better Contextual Awareness! Normal LSTMs process the words only in one direction, and this limits the contextual awareness of the network. Even the Bi-Directional LSTMs learn the context in forward & backward direction and concatenate them, while a better approach would be to look at both the words ahead and back together! Long Sequences! We already know that LSTMs can perform better than RNNs due to their gates, but even with this, the improvement in the case of LSTMs is not significant when we are trying tasks that crunch a large number of sentences, for example — Text Summarization and Questions & Answers.

With Transformers, let’s see how we can address these issues.

Transformers

This architecture was first show-cased in Attention is All You Need! (2017) paper. The complete architecture can be split into two parts — Encoder & Decoder. In the attached image, the structure on the left is Encoder and on the right is Decoder

Encoder-Decoder architecture from Attention is All You Need! (2017) Paper

1. Encoder

In a nutshell, an Encoder tries to understand the context of an input sentence, and in-process learn what the language is! Let’s see how data moves in an encoder.

The input to an Encoder is a sentence — say The big red dog. These individual words are first tokenized and replaced by pre-trained word embeddings in the Input Embeddings layer.

layer. In the next step, we add a Position Encoding to individual word vectors. Essentially a function that maps to the position of the word in a sentence. We add this to our initial input embeddings to make sure that apart from the word, we also have a position aspect of it embedded in, while we pass the sentence into the encoder.

to individual word vectors. Essentially a function that maps to the position of the word in a sentence. We add this to our initial input embeddings to make sure that apart from the word, we also have a position aspect of it embedded in, while we pass the sentence into the encoder. Next, we pass this to a Self-Attention Layer. The idea here to create a matrix of importance each word has while defining context for other words in the same sentence. Each word would obviously have significant importance with itself, and then with other words.

The darker the red, the higher the attention each individual word has with all the other words. — Image from Transformers Explained — https://www.youtube.com/watch?v=TQQlZhbC5ps

And, finally, we pass it through a Feed-Forward layer, which is basically a dense neural network layer.

The biggest advantage of this network is that we don’t need to pass the data sequentially but in parallel! This greatly improves the training time. And since, we are looking at the words in an overall manner, without any directional sense, the overall understanding of language or context is much better than other architectures.

To understand BERT and XLNet, you only need to know Encoders. So, if that is your goal, please jump to the section on BERT. I’ll give explaining the Decoders a shot next.

2. Decoder

In a nutshell, a Decoder takes the output sentence as the input and tries to output the next word. Visualize the whole Transformer architecture as a Language Translation Model — input as “The big red dog” in English and output as “Le Gros Chien rogue” in French. Now let's try to see how this output flows in a decoder network.

The first step is similar to Encoder. Embedding, coupled with Positional Encoding for each output word.

Next, we compute Masked Self-Attention . This is slightly different from before, as here we intentionally mask the words that are yet to come. Essentially we don’t provide the words that are supposed to come next while calculating the importance. Now, this is interesting because this is different from what we did in Encoder. Intuitively, while predicting the next French word, we can use the context of all English words, but we can’t use the next french word, because then the model would just output that french word. I know it’s a little confusing, but it’ll become clear as we see how these models are used in practice.

. This is slightly different from before, as here we intentionally mask the words that are yet to come. Essentially we don’t provide the words that are supposed to come next while calculating the importance. Now, this is interesting because this is different from what we did in Encoder. Intuitively, while predicting the next French word, we can use the context of all English words, but we can’t use the french word, because then the model would just output that french word. I know it’s a little confusing, but it’ll become clear as we see how these models are used in practice. Then, the input from the Decoder network is looked at alongside Encoder’s input. Here, we try to establish the context or relationship between English and French words together, but again with masking for the French words

Finally, as the word travels through the Feed-Forward and Linear layer, we reach the Softmax. The softmax is essentially on all the words in Vocabulary, which gives us a probability score for all the words in the dictionary for the next word. The highest probability word is chosen for each position.

Do note that, again this process happens in parallel, similar to Encoder.

I hope this gives you some insight into the architecture of Transformers, and how they can be an improvement over LSTMs. Let’s try and look at two such architectures & their implementation in detail.

BERT

Bert stands for Bi-directional Encoder Representation from Transformers. As the name implies, this architecture uses the Encoder part of the Transformers network, but the difference is that multiple encoder networks are stacked one after the other. Another important aspect that makes BERT stand out is its training methodology. Let’s try to understand that.

1. Pre-Training

This is the phase where the model understands what is language and context. For this part, BERT uses two simultaneous tasks to train —

Masked Language Modeling — Intuitively, this is like a “fill in the blanks” learning task. The model randomly masks some portion of sentences and its job is to predict those masked words. For example — With input being, “The [MASK] brown fox [MASK] over the lazy dog.”, the output, would be [“quick”, “jumped”].

— Intuitively, this is like a “fill in the blanks” learning task. The model randomly masks some portion of sentences and its job is to predict those masked words. For example — With input being, “The brown fox over the lazy dog.”, the output, would be [“quick”, “jumped”]. Next Sentence Prediction — Here, BERT takes in two sentences and determines if the second sentence follows the first one, basically like a Classification Task.

In a nutshell, these two tasks give BERT an understanding of language and context both within a sentence and over multiple sentences.

2. Fine-Tuning

Now the model basically understands what the language and context are, next is to train it for our specific task. In our case, this would be Sentiment Classification. We simply add Dense layers at the end of the network to get output, as per our specific task.

Let’s start with implementing BERT. We’ll look at the familiar fast.ai construct, however since these models are not directly available in fast.ai, we’ll have to create classes for Tokenizers, Vocabulary, and Numericalizers for fast.ai. This is exceptionally confusing, especially if one is not familiar with these libraries.

Note that the implementation below is inspired by this excellent Kernel Tutorial and Medium Blog by the same author. This not only allows you to implement transformers quickly, but also gives tremendously flexibility in trying out other powerful architectures.

That said, I’ll try to share my interpretation of this tutorial below. Feel free to check out this kernel for better clarity. There are other tutorials regarding the same, which I’ll link below, but if you want to settle on a general solution to load multiple types of Transformers Models, I would recommend reading this implementation.