Step-by-Step BERT Explanation & Implementation Part 1 - Preprocessing
By: Phillip Kim
Tags: NLP, Transformers, Neural Networks, Bert, Deep Learning

In Natural Language Processing (NLP) field, shortage of training data is one of the biggest challenges. Because NLP problems are very diverse and distinct, task-specific datasets lack human-labeled examples. However, deep learning based models benefit much from millions, if not billions, of data to achieve close or par level of human accuracy. To circumvent this lack of data issue, researchers have developed numerous techniques for training general purpose language representation models using enormous amount of unlabeled corpus on the web. This is known as pre-training and pre-trained models can be fine-tuned with much smaller datasets for specific NLP tasks such as question-answering and sentiment analysis. Recent results from many of such tasks have shown huge improvement over the traditional way of training with huge amount of datasets.

In Nov 2018, Google open-sourced a new technique for NLP pre-training called, Bidirectional Encoder Representations from Transformers, or BERT. BERT builds upon previous pre-trained representation — including Semi-supervised Sequence Learning, Generative Pre-Training, ELMo, and ULMFit. Unlike these models, however, BERT is the first deeply bidirectional unsupervised language representation.

Pre-trained representations can be context-free or contextual, and contextual representations can, further, be unidirectional or bidirectional. For instance, in a following sentence, “I would like to sell my investment account”, the word investment means the same as it would in “top of an investment club” in a context-free model. In a unidirectional model, investment is represented based on “I would like to sell my” but not “account”. However, BERT uses both previous and next contexts to represent the word investment. This process starts from the very bottom of a deep neural network, making it deeply bidirectional.

An image below shows a visualization of BERT representation vs. OpenAI GPT and ELMO.

BERT vs OpenAI GPT vs ELMO (Image source: original paper)

Unidirectional models are trained by predicting each word conditioned on the previous words. However, this technique cannot be used for bidirectional models as we cannot condition on from front and back. This would indirectly cause a mirror effect in a multilayer model. To overcome this problem, masking technique is used on random 15% of the data being trained. For example:

Now that we understand why one would implement BERT to solve a task-specific NLP problem, let’s dive right in.

import torch

from torch.utils.data import TensorDataset, DataLoader, RandomSampler, SequentialSampler

from keras.preprocessing.sequence import pad_sequences

from sklearn.model_selection import train_test_split

from tqdm import tqdm, trange

import pandas as pd

import io

import numpy as np

import matplotlib.pyplot as plt

MAX_LEN = 128

First, let’s import necessary libraries and if you don’t have these libraries, just simply do pip install {library name} or conda install {library name} if you are using conda to install them. We set the maximum length of truncation to be 128 length. We will come back to “truncation” later in Part 2. We are using PyTorch for this implementation. For Tensorflow implementation, I will post another blog to cover this.

def bert_id_cat(df):

df['category_id'] = df['Scenario'].factorize()[0]

category_id_df = df[['Scenario', 'category_id']].drop_duplicates().sort_values('category_id')

category_to_id = dict(category_id_df.values)

id_to_category = dict(category_id_df[['category_id', 'Scenario']].values)

return category_to_id, id_to_category

Here, I’m using a dataset that has Scenario column as label column. We need to build a dictionary that converts labels into numerical labels and vise versa for easy conversion.

def save_dict(category_to_id, id_to_category):

f = open("bert_model/category_to_id_bert.pickle", "wb")

f.write(pickle.dumps(category_to_id))

f.close()

f = open("bert_model/id_to_category_bert.pickle", "wb")

f.write(pickle.dumps(id_to_category))

f.close()

Because we want to save the created dictionaries, we define a save function to store these dictionaries.

def bert_preprocess(sentences):

sentences_list = []

for sent in sentences:

sent_find = sent.find("www")

if sent_find != -1:

sent = sent[:sent_find - 1]

sent = pretext.remove_special_characters(str(sent))

sent = pretext.expand_contractions(sent)

sent = "[CLS] " + str(sent) + " [SEP]"

sentences_list.append(sent)

return sentences_list

In order to feed correct form of data into BERT model, we separate each datum starting with [CLS] and ending with [SEP]. In addition, I’m using my own pretext library to clean each sentence. You don’t need this library to preprocess the texts but if you need this file let me know in the comment down below.

def tokenizedTexts(tokenizer, sentences):

tokenized_texts = []

for sent in sentences:

sent = tokenizer.tokenize(sent)

if len(sent) > 512:

sent = sent[:511]

sent.extend(['[SEP]'])

else:

pass

tokenized_texts.append(sent)

return tokenized_texts

We tokenized the preprocessed texts into tokens. PyTorch can only support a sentence of up to the length of 512 tokens, we limit our sentences to 512 tokens.

Now, we are ready to create input ids and start on attention masking process. We will continue this on Part2 of the series.