Machine Learning — Text Classification, Language Modelling using fast.ai | by Javaid Nabi | Towards Data Science

Transfer learning is a technique where instead of training a model from scratch, we reuse a pre-trained model and then fine-tune it for another related task. It has been very successful in computer vision applications. In natural language processing (NLP) transfer learning was mostly limited to the use of pre-trained word embeddings. Research in the field of using language modelling during pre-training have resulted in massive leap in state-of-the-art results for many of the NLP tasks, such as text classification, natural language inference and question-answering through various approaches such as ULMFiT, the OpenAI Transformer, ELMo and Google AI’s BERT.
In this post, we will discuss the limitations of word embedding approach in transfer learning for NLP problems and use of language model to build a text classifier using fast.ai library.
Word embeddings algorithms word2vec and GloVe provide a mapping of words to a high-dimensional continuous vector space where different words with a similar meaning have a similar vector representation. These word embeddings, pre-trained on large amounts of unlabeled data, are used to initialize the first layer of a neural network called embedding layer, the rest of the model is then trained on data of a particular task. This kind of transfer learning in NLP problems is shallow as learning is transferred to only the first layer of the model, the rest of the network still needs to be trained from scratch. A model initialized with word embeddings needs to learn from scratch not only to disambiguate words, but also to derive meaning from a sequence of words. In order to do so, NLP models initialized with these shallow representations require a huge dataset to achieve good performance, which can result in very large computational costs as well [1].
The core objective of a language modelling is of language understanding and it requires modeling complex language phenomena to deal with challenging language understanding problems such as translation, question answering and sentiment analysis. A language model attempts to learn the structure of natural language through hierarchical representations, and thus contains both low-level features (word representations) and high-level features (semantic meaning). A key feature of language modelling is that it is generative, meaning that it aims to predict the next word given a previous sequence of words. It is able to do this because language models are typically trained on very large datasets in an unsupervised manner, and hence the model can “learn” the syntactic features of language in a much deeper way than word embeddings [2].
To predict the next word of a sentence, the model actually needs to know quite a lot about the language and quite a lot of world knowledge. Here is an an example:
“I’d like to eat a hot ___”: Obviously, “dog”, right?
“It was a hot ___”: Probably “day”
As you can see, there’s not enough information here to decide what the next word probably is. But with a neural net, you absolutely can, provided you train a neural net to predict the next word of a sentence then you actually have a lot of information [3].
fastai library is focused on using pre-trained Language Models and fine-tuning them, done in below three steps:
Data pre-processing in a minimum amount of code.Create a language model with pre-trained weights that you can fine-tune to your dataset.Create other models such as classifiers on top of the language model.
Data pre-processing in a minimum amount of code.
Create a language model with pre-trained weights that you can fine-tune to your dataset.
Create other models such as classifiers on top of the language model.
Before we proceed further, we need to setup environment for fast.ai.
To install fastai:
conda install -c pytorch -c fastai fastai pytorch
In case you want to try out some ready to run options:
Colab: Free, needs installation of fast.ai
Kaggle Kernels: Free, No setup required, but not officially supported
Floydhub; No installation required, 2 hours free credit for CPU and 2 for GPU
I used Colab for initial learning but faced lot of ‘disconnected’ issues. Kaggle is another good option as well. Floydhub worked smoothly but after free credits, you need to pay for the usage. Having setup your environment let us proceed for some action.
Let us build a text classifier to classify the sentiments of IMDB movie dataset using the techniques discussed so far. IMDb dataset contains set of 25,000 movie reviews for training, and 25,000 for testing. We already have the IMDb data downloaded and saved in a csv format from the previous post. Let us load the data into a dataframe,
using below piece of code.
df_imdb = pd.read_csv(path/'movies_data.csv')df_imdb.head()#Loading only few training and validation samples, for quick training timetrn_texts = df_imdb.loc[10000:14999, 'review'].valuestrn_labels = df_imdb.loc[10000:14999, 'sentiment'].valuesval_texts = df_imdb.loc[36000:38999, 'review'].valuesval_labels = df_imdb.loc[36000:38999, 'sentiment'].valuesnp.random.seed(42)trn_idx = np.random.permutation(len(trn_texts))val_idx = np.random.permutation(len(val_texts))trn_texts = trn_texts[trn_idx]val_texts = val_texts[val_idx]trn_labels = trn_labels[trn_idx]val_labels = val_labels[val_idx]col_names = ['labels','text']df_trn = pd.DataFrame({'text':trn_texts, 'labels':trn_labels}, columns=col_names)df_val = pd.DataFrame({'text':val_texts, 'labels':val_labels}, columns=col_names)
We are using 5000 [2500 each label] from training and 3000[1500 each label] from validation examples. I am using smaller set to finish the training faster, please use full dataset for improved performance.
Creating a dataset from your raw texts is very simple. The library provides very easy to use API’s [4], depending on how our data is structured, to create a data class TextDataBunch for text processing, from_csv, from_folder,from_df refer to documentation for more details. Here we’ll use the method from_df of the TextLMDataBunch to create a language model specific data bunch:
# Language model datadata_lm = TextLMDataBunch.from_df('./', train_df=df_trn, valid_df=df_val)
This does all the necessary preprocessing behind the scene. Let us see how the data is encoded by fast.ai
data_lm.show_batch()
We can see a number of tags are applied to words as shown above. This is to retain all the information which can be used to gather an understanding of the new task’s vocabulary. All punctuation, hashtags and special characters are also retained. The text is encoded with various tokens like below:
xxbos: Begining of a sentence
xxfld: Represent separate parts of a document like title, summary etc., each one will get a separate field and so they will get numbered (e.g. xxfld 1, xxfld 2).
xxup: If there's something in all caps, it gets lower cased and a token called xxup will get added to it. Words that are fully capitalized, such as “I AM SHOUTING”, are tokenized as “xxup i xxup am xxup shouting“
xxunk: token used instead of an uncommon word.
xxmaj: token indicates that there is capitalization of the word. “The” will be tokenized as “xxmaj the“.
xxrep: token indicates repeated word, if you have 29 ! in a row, (i.e. xxrep 29 !).
Vocabulary: List of unique possible tokens is called the vocabulary. Listing below first 20 unique tokens in order of frequency:
data_lm.vocab.itos[:20]['xxunk', 'xxpad', 'xxbos', 'xxfld', 'xxmaj', 'xxup', 'xxrep', xxwrep', 'the', '.', ',', 'a', 'and', 'of', 'to', 'is', 'it', 'in', i', 'this']
Numericalization: Finally it is easier for machine to deal with the numbers so replace the tokens with the location of the token in the vocab:
The default vocab size is set to 60,000 words and min count for a word to be added to vocab is 2, to avoid getting the weight matrix huge.
Save and Load: We can save the data bunch after the pre-processing is done. We can load as well whenever we need.
# save and loaddata_lm.save('tmp_lm')data_lm = TextClasDataBunch.load('./', 'tmp_lm')
Fast.ai has a pre-trained Wikitext model, consisting of a pre-processed subset of 103 million tokens extracted from Wikipedia. It’s a model that understands a lot about language and a lot about what language describes. Next step is to fine-tune this model and do transfer learning to create a new language model that’s specifically good at predicting the next word of movie reviews.
This is the first stage of training, where we use the pre-trained language model weights and fine-tune it with the training data of IMDb movie reviews. When we create a learner, we have to pass in two things:
The data: our language model data (data_lm)
A pre-trained model: here, the pre-trained model is the Wikitext 103 model that will be downloaded for you from fastai.
# Language modellearner = language_model_learner(data_lm, pretrained_model=URLs.WT103_1, drop_mult=0.5)
drop_mult , a hyper-parameter ,used for regularization, sets the amount of dropout. If the model is over-fitting increase it, if under-fitting, you can decrease the number.
How to fine-tune the pre-trained model on our movie review data? Learning rate hyper-parameter is one of the most important parameters to train a model. Fast.ai provides a convenient utility (learn.lr_find) to search through a range of learning rates to find the optimum one for our dataset. Learning rate finder will increase the learning rate after each mini-batch. Eventually, the learning rate is too high that loss will get worse. Now look at the plot of learning rate against loss and determine the lowest point (around 1e-1 for the plot below) and go back by one magnitude and choose that as a learning rate (something around 1e-2).
Training the model:
We start training the model with learning rate 1e-2 using fit_one_cycle.
fast.ai library uses latest techniques from deep learning research and one cycle learning is from one of the recent paper and turned out to be both more accurate and faster than any previous approach. First argument ‘1’ is number of epoch runs . We get an accuracy of 29% after running just one epoch.
It trained last layers and basically left most of the model exactly as it was. But what we really want is to train the whole model. Normally after we fine-tune the last layers, the next thing we do is we go unfreeze (unfreeze the whole model for training) and train the whole thing.
learn.unfreeze()learn.fit_one_cycle(1, 1e-3)epoch  train_loss  valid_loss  accuracy1      3.897383    3.977569    0.302463
Accuracy = 0.3 means the model is guessing the next word of the movie review correctly about a third of the time. That sounds like a pretty high number. So it’s a good sign that my language model is doing pretty well.
To evaluate our language model, we can now run learn.predict and pass in the start of a sentence and specify the number of words we want it to guess.
That is pretty decent response and looks like correct grammar. After fine-tuning we get a model that’s good at understanding movie reviews and we can fine-tune that with transfer learning to classify movie reviews to be positive or negative. Let us save the encoding of the model to be used later for classification.
learn.save_encoder('fine_enc')
The part of the model that has the understanding of the sentence is called the encoder. So we save it to later use it during the classification stage.
Now we’re ready to create our classifier. Step one, is to create a data bunch, TextClasDataBunch, passing the vocab from the language model to make sure that this data bunch is going to have exactly the same vocab. Batch size bs to be used is according to the GPU memory you have available, for a 16GB GPU around bs=64 will work fine. You can find whatever batch size fits on your card and use it accordingly.
# Classifier model datadata_clas = TextClasDataBunch.from_df('./', train_df=df_trn, valid_df=df_val, vocab=data_lm.train_ds.vocab, bs=32)
Finally we will create a text classifier learner. Load in our pre train model, the encoding part we saved earlier ‘fine_enc’.
# Classifierclassifier = text_classifier_learner(data_clas, drop_mult=0.5)classifier.load_encoder('fine_enc')
Again, we follow the same procedure to find the learning rate and train the model.
The learning rate around 2e-2 seems right, so let us train the classifier:
Wow 85% accuracy in 16 minutes of training and just using 5K training and 3K validation samples. This is the power of transfer learning.
Loss Plot: Let us plot the loss while training the model:
The loss curve seems going down smoothly and has not reached saturation point yet. fastai calculates the exponentially weighted moving average of the losses thus makes it easier to read these charts [by making the curve smoother] at the same time it might be a batch or two behind where they should be.
Let us understand the techniques fast.ai uses underneath for such impressive results.
Discriminative learning rates: Applying different learning rate to layers as you go from layer to layer. When fitting a model you can pass a list of learning rates which will apply a different rate to each layer group. When working with a Learner on which you've called split, you can set hyper-parameters in four ways:
param = [val1, val2 ..., valn] (n = number of layer groups)param = valparam = slice(start,end)param = slice(end)
param = [val1, val2 ..., valn] (n = number of layer groups)
param = val
param = slice(start,end)
param = slice(end)
If we chose to set it in way 1, we must specify a number of values exactly equal to the number of layer groups. If we chose to set it in way 2, the chosen value will be repeated for all layer groups. If you pass slice(start,end) then the first group's learning rate is start, the last is end, and the remaining are evenly spaced.
If you pass just slice(end) then the last group's learning rate is end, and all the other groups are end/10. For instance (for our learner that has 3 layer groups):
learn.lr_range(slice(1e-5,1e-3)), learn.lr_range(slice(1e-3))(array([1.e-05, 1.e-04, 1.e-03]), array([0.0001, 0.0001, 0.001 ]))
The bottom of the slice and the top of the slice is the difference between how quickly the lowest layer of the model learns versus the highest layer of the model learns. As you go from layer to layer, we decrease the learning rate. The lowest levels are given smaller learning rates so as not to disturb the weights much.
What is fit_one_cycle()? It is one cycle of learning rate, start low, go up, and then go down again. Let us plot the learning rate per batch using plot_lr
When we call fit_one_cycle, we are actually passing in a maximum learning rate. The left side plot shows learning rate change vs the batches. The learning starts slow and it increases about half the time and then it decreases about half the time. As you get close to the final answer you need to anneal your learning rate to hone in on it. The motivation behind this is that during the middle of learning when learning rate is higher, the learning rate works as regularization method and keep network from over-fitting. This helps the network to avoid steep areas of loss and land better flatter minima. Please refer to this paper by Leslie smith which talks in great detail about neural network hyper-parameter tuning and you can find most of these ideas implemented in fastai.
There is one more argument(moms=(0.8,0.7))-momentums equals 0.8,0.7. Basically for training recurrent neural networks (RNNs), it really helps to decrease the momentum a little bit.The right side above is the momentum plot. Every time our learning rate is small, our momentum is high. Why is that? Because as you are learning small learning rate, but you keep going in the same direction, you may as well go faster (higher momentum). But as you are learning high learning rate, but you keep going in the same direction, you may overshoot the target, so momentum should be slowed. This trick can help you train 10 times faster.
To improve the accuracy further, fast.ai provides some more tricks;freeze_to. Don't unfreeze the whole thing but to unfreeze one layer at a time. The below approach works very well and gives incredible results.
unfreeze the last two layers freeze_to(-2), train it a little bit more
unfreeze the next layer freeze_to(-3), train it a little bit more
unfreeze the whole thing unfreeze(), train it a little bit more
We reached an accuracy of 90%. The training loss is still higher than the validation loss so we are not over-fitting yet, there is still scope of improving accuracy by running more epochs.
In classification problems it is very useful to use something called a confusion matrix which shows you for each label, how many times was it predicted correctly. Confusion-matrix is good technique to summarize the performance of a classification algorithm.
We use ClassificationInterpretationclass to do the job for us.
Let us use our classifier and predict some movie review:
This is predicted as ‘0’, negative review. Fantastic!!!
We have previously done sentiment classification of the IMDb movie review using classical machine learning approach here and then using word embedding approach here. The language modelling approach using fast.ai is the simplest and most powerful tool, I have come across. The library provides very easy to use methods and with a few lines of code, you can achieve state-of-the-art results. Please refer to jupyter notebook here.
We discussed briefly the use of transfer learning in NLP problems. We explored the fast.ai library and different hyper-parameter tuning techniques in detail. We created a language model and later applied it on the text classification problem. I hope you enjoyed this post and learned something new and useful.
Thank you for reading.