Fastai - Disaster Prediction using ULMFiT | by Sachin Chaturvedi | Towards Data Science

Last month, I started the fastai MOOC(2019) - Deep Learning for Coders. It covers various topics like Computer Vision, Natural Language Processing, Collaborative Filtering, etc. The most fascinating part I found was the application of Transfer Learning in the field of Natural Language Processing. The approached used by fastai is Universal Language Model Fine-tuning (ULMFiT), which is a transfer learning methodology applied in the field of Natural Language Processing.
course.fast.ai
According to Wikipedia, Transfer learning (TL) is a research problem in machine learning (ML) that focuses on storing knowledge gained while solving one problem and applying it to a different but related problem. For example, knowledge gained while learning to recognize cars could apply when trying to recognize trucks.
To show the use case of ULMFiT, I’ll apply the same on the Real or Not? NLP with Disaster Tweets competition on Kaggle. To give you an overview of the data, the data set contains two CSV files train.csv and test.csv representing the training data set and the test data set respectively. The training set contains the tweet data in text column and target value in target column, the value of which is 1 if it is a real disaster or 0 if it is not a real disaster. The test set contains only tweet data and no target values. The task is to predict whether a tweet represents a real disaster or not.
The paragraph given below is taken from Understanding building blocks of ULMFiT:
High level idea of ULMFiT is to train a language model using a very large corpus like Wikitext-103 (103M tokens), then to take this pre-trained model’s encoder and combine it with a custom head model, e.g. for classification, and to do the good old fine tuning using discriminative learning rates in multiple stages carefully. Architecture that ULMFiT uses for it’s language modeling task is an AWD-LSTM. The name is an abbreviation of ASGD Weight-Dropped LSTM.
Refer to this paper written by Jeremy Howard and Sebastian Ruder if you want to read more about ULMFiT.
In Computer Vision problems, transfer learning is used to help in classification directly but in the case of NLP, we first build a language model that basically predicts the next word of a sentence(the model has to understand the language in which the text is written(e.g., English, etc.)) and then build our classification model using the language model’s encoder and vocabulary.
As mentioned in the paper, we will be using AWD-LSTM architecture pre-trained on Wikipedia. We could directly use this pre-trained language model to build our disaster tweet classifier but the gist here is that the English language of Wikipedia would be different from the English language of the tweets. So we will fine-tune our pre-trained language model using the tweet data and then build our classifier on top of that. As explained in the Deep Learning for Coders with Fastai and PyTorch book where they used the same pre-trained architecture on the IMDb Reviews data set to classify if the review is positive or negative. They explain that the IMDb Reviews English is more informal, contains names of movies, directors, actors, etc. than the regular Wikipedia English on which the architecture is pre-trained on.
We import our test.csv and train.csv to get our training data set and test data set. I cannot share the data set as it is from a Kaggle Competition, you can download it yourself by logging into your account and adhering to the competition rules.
# Importing Pandasimport pandas as pd# Importing fastai libraries for text and callbacksfrom fastai.text import *from fastai.callbacks import *train = pd.read_csv('train.csv')test = pd.read_csv('test.csv')
I won’t be going into details on how the language model is built and you should really check out the MOOC and also the Deep Learning for Coders with Fastai and PyTorch book for further reading. The basic idea is that text data cannot be directly fed to the model and it needs to be converted to numbers so that we can apply our mathematical functions to it. This is done using Tokenization and Numericalization. In Tokenization, we convert the text into a list of tokens and in Numericalization we convert them into numbers based on their index. You should refer to the book if you want to dive deeper.
We’ll be using the Data Block API that does the above for us under the hood and then we will feed the databunch created to our language model.
data_lm = (TextList.from_df(pd.concat([train[['text']], test[['text']]], ignore_index=True, axis=0))           .split_by_rand_pct(0.15)           .label_for_lm()           .databunch(bs=128))data_lm.show_batch()
We ignore the labels and take the text corpus from both training and test data. Remember that we are making a language model and not a classification model right now. We are just including text data as much as we can for our language model to predict the next word of a sentence. Next, we use our data_lm databunch to make our language model.
learn = language_model_learner(data_lm, AWD_LSTM, drop_mult = 0.5)learn.lr_find()learn.recorder.plot(suggestion = True)
learn.fit_one_cycle(1, 1e-2, moms=(0.8,0.7))
We’ll be unfreezing the model and we’ll fit more. We’ll use callbacks to select the best model.
Better model found at epoch 0 with accuracy value: 0.4097544550895691.Better model found at epoch 1 with accuracy value: 0.4404464364051819.Better model found at epoch 2 with accuracy value: 0.4609375.Better model found at epoch 3 with accuracy value: 0.47495537996292114.Better model found at epoch 4 with accuracy value: 0.48810267448425293.Better model found at epoch 5 with accuracy value: 0.49515628814697266.Better model found at epoch 6 with accuracy value: 0.4975222945213318.Better model found at epoch 9 with accuracy value: 0.49756699800491333.
We’ll select the best accuracy which is at epoch number 9. We’ll then save the language model and the encoder.
learn.save('fine_tuned')learn.save_encoder('fine_tuned_enc')
Next, we’ll make our classifier. For that, we’ll need to create a new databunch. We’ll take the validation set as 10% and we’ll keep our vocabulary same as the vocabulary of the language databunch. We’ll also add our test data in the separate add_test parameter.
data_clas = (TextList.from_df(df, vocab=data_lm.vocab)             .split_by_rand_pct(0.1)             .label_from_df('target')             .add_test(TextList.from_df(test['text'], vocab=data_lm.vocab))             .databunch(bs=128))
We’ll build the classifier using the same encoder as our language model.
learn = text_classifier_learner(data_clas, AWD_LSTM, drop_mult=0.5, metrics=[accuracy, FBeta(beta=1)])learn.load_encoder('fine_tuned_enc')
We’ll do a lr_find() check and then plot to see the graph.
Let’s fit one cycle. We see that we get an accuracy of 77.66%.
learn.fit_one_cycle(1, 1e-3, moms=(0.8,0.7))
Let’s unfreeze the last 2 layers and train for one cycle. Our accuracy increases to 79.5%.
learn.freeze_to(-2)learn.fit_one_cycle(1, slice(1e-3/(2.6**4),1e-2), moms=(0.8,0.7))
We’ll unfreeze the last 3 layers now and train again for one more cyle. Our accuracy increases to 81.73%!
learn.freeze_to(-3)learn.fit_one_cycle(1, slice(5e-3/(2.6**4),5e-3), moms=(0.8,0.7))
What we used here were discriminative learning rates which were introduced in ULMFiT. As explained in the article 10 New Things I Learnt from fast.ai v3:
Discriminative learning rates for pre-trained modelsTrain earlier layer(s) with super low learning rate, and train later layers with higher learning rate. The idea is to not drastically alter the almost-perfect pre-trained weights except for minuscule amounts, and to be more aggressive with teaching the layers near the outputs. Discriminative learning rate was introduced in ULMFiT.
We’ll unfreeze all layers, train, and use callbacks to select our best model.
callbacks = SaveModelCallback(learn,monitor="accuracy", mode="max", name="best_classification_model")
Better model found at epoch 0 with accuracy value: 0.8160315155982971.Better model found at epoch 1 with accuracy value: 0.8173456192016602.Better model found at epoch 2 with accuracy value: 0.822601854801178.Better model found at epoch 9 with accuracy value: 0.8239158987998962.
We get an accuracy of 82.39%!!
In Transfer Learning we use knowledge from a source and apply it to our target. Implementation of the same in the field of Natural Language Processing has provided an extraordinary state of the art results with minimal training as we use a pre-trained network. You can check out my notebook on Kaggle if you would like to see the code in action: NLP - Disaster Prediction ULMFiT
Note: I have used fastai V1 here. The fastai V2 along with the new MOOC came out on 21st August. Check that out here