Sentence Correctness classifier using Transfer Learning with Huggingface BERT | by Ayush Chaurasia | Towards Data Science

Learn To build production level NLP applications simply using huggingface
Here’s the Link to live Dashboard
In this article, we’ll build a near state of the art sentence classifier leveraging the power of recent breakthroughs in the field of Natural Language Processing. We’ll focus on application of transfer learning in NLP to create high performance models with minimal effort on a range of NLP tasks.
In the last two years or so, NLP research has been quite rapid as compared to other domains of applied deep learning. models like Allen AI’s ELMO, OpenAI’s Open-GPT, and Google’s BERT allowed researchers to smash multiple benchmarks with minimal fine tuning for specific tasks. As a result, NLP research reproduction and experimentation has become more accessible.
BERT (Bidirectional Encoder Representations from Transformers), released in late 2018 by Google researchers is the model we’ll use to train our sentence classifier.
We’ll use transfer learning on the pre-trained BERT model. This approach is better than training a deep model like GRU or LSTM from scratch as:
the pre-trained BERT model weights already encode a lot of information about our language. As a result, it takes much less time to train our fine-tuned model. We’ll train our model with hyper-parameter sweeps to find the best combination within an hour on colab while training a single deep RNN model from scratch would take more than hundreds of hours on GPU!
As we are essentially performing transfer learning, we need significantly less data to build an accurate system. Compared to millions of data points required to train a model from scratch, we can accomplish the same task with only a few thousand data points.
BERT, based on transformer network, is Bi-directional network that produces the language encoding for various tasks such as modelling, classification etc.
Google said BERT helps better understand the nuances and context of words in searches and better match those queries with more relevant results. It is also used for featured snippets. Here’s an example.
Let’s look at another exciting application of this model, i.e , sentence classification.
Now, we’ll quickly move into training and experimentation, but if you want more details about theenvironment and datasets, check out this tutorial by Chris McCormick.
Let’s first install the huggingface library on colab:
!pip install transformers
This library comes with various pre-trained state of the art models. For our sentence classification we’ll use BertForSequenceClassification model.
We’ll use The Corpus of Linguistic Acceptability (CoLA) dataset for single sentence classification. It’s a set of sentences labeled as grammatically correct or incorrect. It was first published in May of 2018, and is one of the tests included in the “GLUE Benchmark” on which models like BERT are competing.
Let’s download and unzip the dataset
If everything executes without errors, you’ll get the following files:
As you can see that we already have a tokenized version of the dataset, but we need to tokenize it again because, in order to apply the pre-trained BERT, we must use the tokenizer provided by the model. This is because:
the model has a specific, fixed vocabulary and
the BERT tokenizer has a particular way of handling out-of-vocabulary words.
Let’s take a look at the format of the dataset by parsing it using pandas
A few things to notice here:
We just have 8551 data points using which train a SOTA deep model by using transfer learning.
The attributes that we care about are sentence and it label .
The acceptability judgement where 0=unacceptable and 1=acceptable.
Finally, let’s get a list of these sentences.
As mentioned earlier, the sentences that are to be fed into the BERT model must be tokenized using the BERT tokenizer. Let’s take a look at an example.
Using this tokenizer on a sentence would result into something like this:Original:  Our friends won't buy this analysis, let alone the next one we propose.Tokenized:  ['our', 'friends', 'won', "'", 't', 'buy', 'this', 'analysis', ',', 'let', 'alone', 'the', 'next', 'one', 'we', 'propose', '.']Token IDs:  [2256, 2814, 2180, 1005, 1056, 4965, 2023, 4106, 1010, 2292, 2894, 1996, 2279, 2028, 2057, 16599, 1012]
Before we process the entire dataset using this tokenizer, there are a few conditions that we need to satisfy in order to setup the training data for BERT:
Add special tokens to the start and end of each sentence. At the end of every sentence, we need to append the special [SEP] token and for classification tasks, we must prepend the special [CLS] token to the beginning of every sentence.
Pad & truncate all sentences to a single constant length
Explicitly differentiate real tokens from padding tokens with the “attention mask”. The “Attention Mask” is simply an array of 1s and 0s indicating which tokens are padding and which aren’t
Here’s how the network architecture will operate on a particular input after all the pre-processing.
The first task is to decide the maximum length of a sentence.
Output
Max sentence length:  47
To follow a general convention that the sizes should be in powers of 2, we’ll choose the closest number that is a power of 2, i.e, 64.
Now, we’re ready to perform the real tokenization. But as we’re using transformers, we can use an inbuilt function tokenizer.encode_plus which automates all of the following tasks:
Split the sentence into tokens.Add the special [CLS] and [SEP] tokens.Map the tokens to their IDs.Pad or truncate all sentences to the same length.Create the attention masks which explicitly differentiate real tokens from [PAD] tokens.
Split the sentence into tokens.
Add the special [CLS] and [SEP] tokens.
Map the tokens to their IDs.
Pad or truncate all sentences to the same length.
Create the attention masks which explicitly differentiate real tokens from [PAD] tokens.
We’ll split the the data into train and test set. Divide up our training set to use 90% for training and 10% for validation.
The next steps require us to guess various hyper-parameter values. We’ll automate that taks by sweeping across all the value combinations of all parameters. For doing that, we’ll initialize a wandb object before starting the training loop. The hyper-parameter value for the current run is saved in wandb.config.parameter_name .
Here, we’re using RandomSampler to sample trainset and SequentialSampler to sample validation set.
We’ll be using pre-trained BertForSequenceClassification. We’ll add a single dense or fully-connected layer to perform the task of binary classification. We’ll separate each part of the program as a separate function block.
The num_labels parameter describes the number of final output neurons.
We’ll use an implementation of Adam optimizer with am inbuilt weight-decay mechanism provided by huggingface. The learning rate will be provided by wandb.config
We’ll also initialize a learning rate scheduler to perform learning rate decay. The training epochs is also a hyper-parameter so we’ll initialize that using wandb.config .
There’s only one step left before we train our model. We’ll setup a configuration file that’ll list out all the values a hyper-parameter can take. Then we’ll initialize our wandb sweep agent to log, compare and visualize the performance of each combination.
The metric we’re looking to maximize is the val_accuracy which we’ll log in the training loop.
In the BERT paper, the authors described the best set of hyper-parameters to perform transfer learning and we’re using that same sets of values for our hyper-parameters.
Now we have a sweep id which is required to run a sweep. We now only need a train function that will be repeatedly invoked for each sweep.
On running a wandb sweep, you get a few automatic visualizations along with the ones that we’ve logged manually. It is now time to make deductions.
Some of the most essential information about the model performance can be deduced directly from the parameter correlation graph.
Here, we can see how all of the runs performed in the task of maximizing the validation accuracy. We can deduce the best hyper-parameter combination is batch size of 16, learning rate of 3e-5 and trained for 3 epochs which will result in accuracy of about 84%.
If you want a simpler visualization that just compares the time taken by each run as well as how it performs in optimizing the desired metric, you can refer another useful visualization in the dashboard which compares the validation accuracy Vs the time of a particular run.
Now let’s take a look at the visualizations of the metrics that we logged.
Now you have the BERT trained on best set of hyper-parameter values for performing sentence classification along with various statistical visualizations to support choice of parameters.
This post demonstrates that with a pre-trained BERT model you can quickly create a model with minimum fine-tuning and data using the huggingface interface, depending on the specific NLP task you are interested in.
I hope this has been nice read. All of the results in this article can be reproduced using this colab notebook and the visualizations comparing various run can be viewed in this wandb sweeps page.