Build a Natural Language Classifier With Bert and Tensorflow
By: James Briggs
Tags: Programming, Machine Learning, Python, Data Science, AI

The dataset is full of these duplicates, or phrase segments. We can remove these by typing:

Tokenization

We have our text data in the text column, which we now need to tokenize. We will use the BERT tokenizer, as we will be using a BERT transformer later.

Here we are first importing the transformers library and initializing a tokenizer for the bert-base-cased model used. A list of models can be found here. We then define a function tokenize that handles tokenization.

We use the encode_plus method of our BERT tokenizer to convert a sentence into input_ids and attention_mask tensors.

Xids and Xmask are our complete input_ids and attention_mask tensors respectively.

The input IDs are a list of integers that are uniquely tied to a specific word.

The attention mask is a list of 1s and 0s which correspond to the IDs in the input IDs array — BERT reads this and only applies attention to IDs that correspond to an attention mask value of 1. This allows us to avoid applying attention to padding tokens.

Our encode_plus arguments are:

Our sentence . This is simply a string representing one tweet.

. This is simply a string representing one tweet. The max_length of our encoded outputs. We use a value of 32 which means that every output tensor has a length of 32.

of our encoded outputs. We use a value of which means that every output tensor has a length of 32. We cut sequences that have a length of more than 32 tokens with truncation=True .

. For sequences that are shorter than 32 tokens, we pad them with zeros up to a length of 32 using padding='max_length' .

. BERT uses several special tokens, to mark the start/end of sequences, for padding, unknown words, and mask words. We add those using add_special_tokens=True .

. BERT also takes two inputs, the input_ids and attention_mask . We extract the attention mask with return_attention_mask=True .

and . We extract the attention mask with . By default, the tokenizer will return a token type IDs tensor — which we don’t need, so we use return_token_type_ids=False .

. Finally, we are using TensorFlow, so we return TensorFlow tensors using return_tensors='tf' . If using PyTorch, use return_tensors='pt' .

At the end of this, we return our two tensors. We initialize our two Xids and Xmask arrays beforehand and then populated them with our encoded tensors using a simple for-loop — applying tokenizer to each sentence of our dataset.

Data prep

We now need to prepare our data for training. For this, we will one-hot encode our target labels, creating a simple two-output vector for each sample where [1, 0] represents negative sentiment and [0, 1] positive.

One-hot encoding. Before (top) and after (bottom).

This whole process can take some time. I like to save the encoded arrays so that we can pick up from here if anything goes wrong or for future tests.

Now we have all of the encoded arrays, we load them into a TensorFlow dataset object. Using the dataset, we easily restructure, shuffle, and batch the data.

Train-validation split

The final step before training our model is to split our dataset into training, validation, and (optionally) test sets. We will stick with a simple 90–10 train-validation split here.

Model definition

Our data is now ready and we can define our model architecture. We will use BERT, followed by an LSTM layer, and some simple NN layers. Those final layers following BERT are our classifier.

Our classifier consumes the output hidden state tensors from BERT — using them to predict whether we’re seeing something with positive or negative sentiment.

Training

We can now train our model. First, we set up our optimizer (Adam), loss function, and accuracy metric. Then, we compile the model and train!