Text Classification With BERT Model
By: Kamal Khumar
Tags: Bert, Text Classification, Ktrain, Advantages Of Bert

This blog explains, what is BERT, advantages of BERT, how to classify the text using BERT model with Ktrain package.

What is BERT?

BERT, which stands for Bidirectional Encoder Representations from Transformers developed by researchers at Google in 2018, is based on Transformers, a deep learning model in which every output element is connected to every input element, and the weightings between them are dynamically calculated based upon their connection. (In NLP, this process is called attention.)

Historically, language models could only read text input sequentially — either left-to-right or right-to-left — but couldn’t do both at the same time. BERT is different because it is designed to read in both directions at once. This capability, enabled by the introduction of Transformers, is known as bi-directionality.

Why BERT NLP Will Be Revolutionary?

What makes it’s unique from the rest of the model is that it’s is the first deeply bidirectional, unsupervised language representation, pre-trained using only a plain text corpus. Since it’s open-sourced, anyone with machine learning knowledge can easily build an NLP model without the need for sourcing massive datasets for training the model thus saving time, energy, knowledge and resources.

Finally, BERT is pre-trained on a large corpus of unlabelled text which includes the entire Wikipedia (that’s about 2,500 million words) and a book corpus (800 million words).

To accomplish this, we will be using ktrain, a fastai-like interface to Keras. ktrain is open-source and available here.

To install ktrain, simply type the following:

pip3 install ktrain

To begin, let’s import the ktrain and ktrain.text modules:

import ktrain

from ktrain import text

Step 1: Load the data

Load the in-built 20newsgroups data

Step 2: Load BERT and wrap it in a Learner object

The first argument to get_learner uses the ktrain text_classifier function to load the pre-trained BERT model with a randomly initialized final Dense layer. The second and third arguments are the training and validation data, respectively. The last argument get_learner is the batch size. We use a small batch size of 6.

Step 3: Train the model

To train the model, we use the fit_onecycle method of ktrain which employs a 1cycle learning rate policy that linearly increases the learning rate for the first half of training and then decreases the learning rate for the latter half:

As can be seen from the Keras output, this achieves a 99.65% accuracy in four epochs:

begin training using onecycle policy with max lr of 2e-05...

Train on 2257 samples

Epoch 1/4

2257/2257 [==============================] - 498s 221ms/sample - loss: 0.6748 - accuracy: 0.7288

Epoch 2/4

2257/2257 [==============================] - 477s 211ms/sample - loss: 0.1240 - accuracy: 0.9575

Epoch 3/4

2257/2257 [==============================] - 476s 211ms/sample - loss: 0.0394 - accuracy: 0.9867

Epoch 4/4

2257/2257 [==============================] - 475s 211ms/sample - loss: 0.0101 - accuracy: 0.9965

<tensorflow.python.keras.callbacks.History at 0x7fc74aab2588>

Step 4: Plot the learning rate

And the plot will be,

Step 5: Validating the test data

The test data will be validated and the output is,

precision recall f1-score support



alt.atheism 0.91 0.93 0.92 319

comp.graphics 0.99 0.95 0.97 389

sci.med 0.97 0.96 0.97 396

soc.religion.christian 0.96 0.97 0.97 398



accuracy 0.96 1502

macro avg 0.96 0.96 0.96 1502

weighted avg 0.96 0.96 0.96 1502 array([[298, 0, 9, 12],

[ 15, 371, 2, 1],

[ 7, 4, 382, 3],

[ 9, 1, 1, 387]])

Step 6: Storing the model

The full source code available on GitHub.