Text Classification with BERT and Tensorflow in Ten Lines of Code | by Shuyi Wang | Towards Data Science

We all know BERT is a compelling language model which has already been applied to various kinds of downstream tasks, such as Sentiment Analysis and Question answering (QA). In some of them, it over-performed human beings!
Have you ever tried it on text binary classification?
Honestly, till recently, my answer was still NO.
Why not?
Because the example code on BERT’s official GitHub repo was not very user-friendly.
Firstly, I want an IPython Notebook, instead of a Python script file, for I want to get instant feedback when I run a code chunk. Of course, a Google Colab Notebook would be better, for I can use the code right away with the free GPU/TPU.
Secondly, I don’t want to know the detail except for the ones I care. I want to control the useful parameters, such as the number of epochs and batch size. However, do I need to know all the “processors,” “flags” and logging functions?
I am a spoiled machine learning user after I tried all other friendly frameworks.
For example, in Scikit-learn, if you try to build a tree classifier, here is (almost) all your code.
from sklearn.datasets import load_irisfrom sklearn import treeiris = load_iris()clf = tree.DecisionTreeClassifier()clf = clf.fit(iris.data, iris.target)
If you want to do image classification in fast.ai, you need to input these lines.
!git clone https://github.com/wshuyi/demo-image-classification-fastai.gitfrom fastai.vision import *path = Path("demo-image-classification-fastai/imgs/")data = ImageDataBunch.from_folder(path, test='test', size=224)learn = cnn_learner(data, models.resnet18, metrics=accuracy)learn.fit_one_cycle(1)interp = ClassificationInterpretation.from_learner(learn)interp.plot_top_losses(9, figsize=(8, 8))
Not only you can get the classification result, but an activation map as well.
Why on earth cannot Google Developers give us a similar interface to use BERT for text classification?
On Monday, I found this Colab Notebook. It’s an example of predicting sentiment of movie reviews.
I was so excited, for I learned BERT is now included in Tensorflow Hub.
However, when I opened it, I found there are still too many details for a user who only cares about the application of text classification.
So I tried to refactor the code, and I made it.
However, originally, there were still a lot of codes in the notebook.
So I asked my readers to help me package them. Soon, Yan Sun, a CS major graduate student in Huazhong University of Science & Technology did that, and made it a PYPI package named bert-text.
Please follow this link and you will see the IPynb Notebook file on github.
Click the “Open in Colab” Button. Google Colab will be opened automatically.
You need to save a copy to your own Google Drive by clicking on the “COPY TO DRIVE” button.
You only need to do four things after that.
Install the necessary packages.Prepare the data in Pandas Data frame format. I guess it’s easy for most deep learning users.Adjust four parameters if necessary.Run the notebook and get your result displayed.
Install the necessary packages.
Prepare the data in Pandas Data frame format. I guess it’s easy for most deep learning users.
Adjust four parameters if necessary.
Run the notebook and get your result displayed.
Let us install bert-text package and load the API.
!pip install bert-textfrom bert_text import run_on_dfs
My example is a sample dataset of IMDB reviews. It contains 1000 positive and 1000 negative samples in training set, while the testing set contains 500 positive and 500 negative samples.
import pickle!wget https://github.com/wshuyi/info-5731-public/raw/master/imdb-sample.picklewith open("imdb-sample.pickle", 'rb') as f:    train, test = pickle.load(f)
I used it in my INFO 5731 class at UNT to let students compare the result of textblob package, Bag of Words model, simple LSTM with word embedding, and ULMfit.
Now I can add BERT into the list, finally.
You need to run the following line to make sure the training data is shuffled correctly.
train = train.sample(len(train))
Your dataset should be stored in Pandas Data Frame. There should be one training set, called train and one testing set, called test.
Both of them should at least contain two columns. One column is for the text, and the other one is for the binary label. It is highly recommended to select 0 and 1 as label values.
Now that your data is ready, you can set the parameters.
myparam = {        "DATA_COLUMN": "text",        "LABEL_COLUMN": "sentiment",        "LEARNING_RATE": 2e-5,        "NUM_TRAIN_EPOCHS":10    }
The first two parameters are just the name of columns of your data frame. You can change them accordingly.
The third parameter is the learning rate. You need to read the original paper to figure out how to select it wisely. Alternatively, you can use this default setting.
The last parameter is to set how many epochs you want BERT to run. I chose 10 here, for the training dataset is very small, and I don’t want it overfits.
By default, the pre-trained language model is “bert_uncased_L-12_H-768_A-12". It has been trained on English Wikipedia and the BookCorpus. Apart from that, there are five more options you can choose from by now.
bert_multi_cased_L-12_H-768_A-12
bert_uncased_L-24_H-1024_A-16
bert_cased_L-12_H-768_A-12
bert_chinese_L-12_H-768_A-12
bert_cased_L-24_H-1024_A-16
If you want to change to another pre-trained model, say Multilingual one to handle French or German, change the myparam setting as follow:
myparam = {        "DATA_COLUMN": "comment",        "LABEL_COLUMN": "sentiment",        "LEARNING_RATE": 2e-5,        "NUM_TRAIN_EPOCHS":3,        "bert_model_hub":"https://tfhub.dev/google/bert_multi_cased_L-12_H-768_A-12/1"    }
The Chinese language is a particular case. You may need to set myparam like this:
myparam = {        "DATA_COLUMN": "comment",        "LABEL_COLUMN": "sentiment",        "LEARNING_RATE": 2e-5,        "NUM_TRAIN_EPOCHS":3,        "bert_model_hub":"https://tfhub.dev/google/bert_chinese_L-12_H-768_A-12/1"    }
If you want to see the whole output (Optional), you should add the following two lines of code.
import tensorflow as tftf.logging.set_verbosity(tf.logging.INFO)
Okay. Now you can run BERT!
result, estimator = run_on_dfs(train, test, **myparam)
Warning! This line takes you some time to run.
When it’s finished, you can run the last line to get evaluation result of your classification model (on BERT).
result
For such a small training set, I think the result is quite good.
That’s all.
Now you can use the state of the art language modeling technique to train your text binary classifier too!
If you are interested in this blog article, you may also want to read the following ones:
How to Practice Python with Google Colab?
How to Predict Severe Traffic Jams with Python and Recurrent Neural Networks?
Deep Learning with Python and fast.ai, Part 1: Image classification with pre-trained model
Deep Learning with Python and fast.ai, Part 2: NLP Classification with Transfer Learning