A Beginner’s Guide to Using BERT for the First Time | by Arfinda Ilmania | Towards Data Science

BERT has become a new standard for Natural Language Processing (NLP). It achieved a whole new state-of-the-art on eleven NLP task, including text classification, sequence labeling, question answering, and many more. Even better, it can also give incredible results using only a small amount of data. BERT was first released in 2018 by Google along with its paper: BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding.
Now we can easily apply BERT to our model by using Huggingface (🤗) Transformers library. The library already provided complete documentation about other transformers models too. You can check it here. In this post, I will try to summarize some important points which we will likely use frequently. We will take a look at how to use and train models using BERT from 🤗 Transformers. Later, you can also utilize other transformers models (such as XLM, RoBERTa, XLM RoBERTa (my favorite!), BART, and many others) by simply changing a single line of code.
Text classification seems to be a pretty good start to get to know BERT. There are many kinds of text classification tasks, but we will choose sentiment analysis in this case. Here are 5 main points which we will be covered in this post:
InstallationPipelineFine-tuneUsing custom datasetHyperparameter search
Installation
Pipeline
Fine-tune
Using custom dataset
Hyperparameter search
As stated on their website, to run 🤗 Transformers you will need to have some requirement as follow:
Python 3.6+
Pytorch 1.10+ or Tensorflow 2.0
They also encourage us to use virtual environments to install them, so don’t forget to activate it first.
The installation is quite easy, when Tensorflow or Pytorch had been installed, you just need to type:
pip install transformers
In this post, we are going to use Pytorch. But it should be easy if you want to translate it into Tensorflow, just add ‘TF’ at the beginning of each model class name.
When you just want to test or simply use it to predict some sentences, you can use pipeline(). Besides text classification, they already provided many different tasks such as text generation, question answering, summarization, and so on. To run sentiment analysis task, simply type:
from transformers import pipelineclassifier = pipeline('sentiment-analysis')result = classifier('We are very happy to show you the 🤗 Transformers library.')
It uses a model named “distilbert-base-uncased-finetuned-sst-2-english” by default. We can also change to other models that we can find in the model hub. For example, if we want to use nlptown/bert-base-multilingual-uncased-sentiment, then simply do the following:
classifier = pipeline(‘sentiment-analysis’, model=”nlptown/bert-base-multilingual-uncased-sentiment”)
First thing first, we need a dataset. At this point, we are going to use the dataset provided by 🤗 Datasets. They provide a wide range of task options, varying from text classification, token classification, language modeling, and many more. To install it, simply execute the following line:
pip install datasets
We are going to use sst2 dataset from GLUE task and bert-base-uncased pretrained. By runningload_dataset and load_metric, we are downloading dataset as well as metric. load_metricautomatically loads a metric associated with the chosen task.
# ! pip install git+https://github.com/huggingface/transformers.git
# ! pip install git+https://github.com/huggingface/datasets.git

GLUE_TASKS = ["cola", "mnli", "mnli-mm", "mrpc", "qnli", "qqp", "rte", "sst2", "stsb", "wnli"]
task = "sst2"
model_checkpoint = "bert-base-uncased"
batch_size = 16

from datasets import load_dataset, load_metric
actual_task = "mnli" if task == "mnli-mm" else task
dataset = load_dataset("glue", actual_task)
metric = load_metric('glue', actual_task)

Reusing dataset glue (/root/.cache/huggingface/datasets/glue/sst2/1.0.0/7c99657241149a24692c402a5c3f34d4c9f1df5ac2e4c3759fadea38f6cb29c4)

dataset

DatasetDict({
    train: Dataset({
        features: ['sentence', 'label', 'idx'],
        num_rows: 67349
    })
    validation: Dataset({
        features: ['sentence', 'label', 'idx'],
        num_rows: 872
    })
    test: Dataset({
        features: ['sentence', 'label', 'idx'],
        num_rows: 1821
    })
})
dataset["train"][0]

{'idx': 0,
 'label': 0,
 'sentence': 'hide new secretions from the parental units '}
metric

Metric(name: "glue", features: {'predictions': Value(dtype='int64', id=None), 'references': Value(dtype='int64', id=None)}, usage: """
Compute GLUE evaluation metric associated to each GLUE dataset.
Args:
    predictions: list of predictions to score.
        Each translation should be tokenized into a list of tokens.
    references: list of lists of references for each translation.
        Each reference should be tokenized into a list of tokens.
Returns: depending on the GLUE subset, one or several of:
    "accuracy": Accuracy
    "f1": F1 score
    "pearson": Pearson Correlation
    "spearmanr": Spearman Correlation
    "matthews_correlation": Matthew Correlation
""", stored examples: 0)
import numpy as np

fake_preds = np.random.randint(0, 2, size=(64,))
fake_labels = np.random.randint(0, 2, size=(64,))
metric.compute(predictions=fake_preds, references=fake_labels)

{'accuracy': 0.546875}
import datasets
import random
import pandas as pd
from IPython.display import display, HTML

def show_random_elements(dataset, num_examples=10):
    assert num_examples <= len(dataset), "Can't pick more elements than there are in the dataset."
    picks = []
    for _ in range(num_examples):
        pick = random.randint(0, len(dataset)-1)
        while pick in picks:
            pick = random.randint(0, len(dataset)-1)
        picks.append(pick)
    
    df = pd.DataFrame(dataset[picks])
    for column, typ in dataset.features.items():
        if isinstance(typ, datasets.ClassLabel):
            df[column] = df[column].transform(lambda i: typ.names[i])
    display(HTML(df.to_html()))

show_random_elements(dataset["train"])

To preprocess, we need to instantiate our tokenizer using AutoTokenizer (or other tokenizer class associated with the model, eg: BertTokenizer). By calling from_pretrained(), we download the vocab used during pretraining the given model (in this case, bert-base-uncased). The vocab is useful so that the tokenization results are corresponding to the model’s vocab.
from transformers import AutoTokenizer
    
tokenizer = AutoTokenizer.from_pretrained(model_checkpoint, use_fast=True)

tokenizer("Hello, this one sentence!", "And this sentence goes with it.")

{'input_ids': [101, 7592, 1010, 2023, 2028, 6251, 999, 102, 1998, 2023, 6251, 3632, 2007, 2009, 1012, 102], 'token_type_ids': [0, 0, 0, 0, 0, 0, 0, 0, 1, 1, 1, 1, 1, 1, 1, 1], 'attention_mask': [1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1]}
task_to_keys = {
    "cola": ("sentence", None),
    "mnli": ("premise", "hypothesis"),
    "mnli-mm": ("premise", "hypothesis"),
    "mrpc": ("sentence1", "sentence2"),
    "qnli": ("question", "sentence"),
    "qqp": ("question1", "question2"),
    "rte": ("sentence1", "sentence2"),
    "sst2": ("sentence", None),
    "stsb": ("sentence1", "sentence2"),
    "wnli": ("sentence1", "sentence2"),
}

sentence1_key, sentence2_key = task_to_keys[task]

def preprocess_function(examples):
    if sentence2_key is None:
        return tokenizer(examples[sentence1_key], truncation=True)
    return tokenizer(examples[sentence1_key], examples[sentence2_key], truncation=True)

preprocess_function(dataset['train'][:5])

{'input_ids': [[101, 5342, 2047, 3595, 8496, 2013, 1996, 18643, 3197, 102], [101, 3397, 2053, 15966, 1010, 2069, 4450, 2098, 18201, 2015, 102], [101, 2008, 7459, 2049, 3494, 1998, 10639, 2015, 2242, 2738, 3376, 2055, 2529, 3267, 102], [101, 3464, 12580, 8510, 2000, 3961, 1996, 2168, 2802, 102], [101, 2006, 1996, 5409, 7195, 1011, 1997, 1011, 1996, 1011, 11265, 17811, 18856, 17322, 2015, 1996, 16587, 2071, 2852, 24225, 2039, 102]], 'token_type_ids': [[0, 0, 0, 0, 0, 0, 0, 0, 0, 0], [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0], [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0], [0, 0, 0, 0, 0, 0, 0, 0, 0, 0], [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]], 'attention_mask': [[1, 1, 1, 1, 1, 1, 1, 1, 1, 1], [1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1], [1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1], [1, 1, 1, 1, 1, 1, 1, 1, 1, 1], [1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1]]}
encoded_dataset = dataset.map(preprocess_function, batched=True)

HBox(children=(FloatProgress(value=0.0, max=68.0), HTML(value='')))
HBox(children=(FloatProgress(value=0.0, max=1.0), HTML(value='')))
HBox(children=(FloatProgress(value=0.0, max=2.0), HTML(value='')))
Fortunately, they also provide a simple interface called Trainer() which makes the training and evaluation process much easier without losing its flexibility to modify a wide range of training options.
First, instantiate and download the model with from_pretrained(). Since our task is sequence classification, we can use AutoModelForSequenceClassification (or other model class associated to the pretrained, eg: BertForSequenceClassification).
We need to define our own compute_metrics function if we want to have other metrics in addition to the loss. This function can be passed to the trainer.
from transformers import AutoModelForSequenceClassification, TrainingArguments, Trainer

num_labels = 3 if task.startswith("mnli") else 1 if task=="stsb" else 2
model = AutoModelForSequenceClassification.from_pretrained(model_checkpoint, num_labels=num_labels)

Some weights of the model checkpoint at bert-base-uncased were not used when initializing BertForSequenceClassification: ['cls.predictions.bias', 'cls.predictions.transform.dense.weight', 'cls.predictions.transform.dense.bias', 'cls.predictions.decoder.weight', 'cls.seq_relationship.weight', 'cls.seq_relationship.bias', 'cls.predictions.transform.LayerNorm.weight', 'cls.predictions.transform.LayerNorm.bias']
- This IS expected if you are initializing BertForSequenceClassification from the checkpoint of a model trained on another task or with another architecture (e.g. initializing a BertForSequenceClassification model from a BertForPreTraining model).
- This IS NOT expected if you are initializing BertForSequenceClassification from the checkpoint of a model that you expect to be exactly identical (initializing a BertForSequenceClassification model from a BertForSequenceClassification model).
Some weights of BertForSequenceClassification were not initialized from the model checkpoint at bert-base-uncased and are newly initialized: ['classifier.weight', 'classifier.bias']
You should probably TRAIN this model on a down-stream task to be able to use it for predictions and inference.

metric_name = "pearson" if task == "stsb" else "matthews_correlation" if task == "cola" else "accuracy"

args = TrainingArguments(
    "test-glue",
    evaluation_strategy = "epoch",
    learning_rate=2e-5,
    per_device_train_batch_size=batch_size,
    per_device_eval_batch_size=batch_size,
    num_train_epochs=1,
    weight_decay=0.01,
    load_best_model_at_end=True,
    metric_for_best_model=metric_name,
)

def compute_metrics(eval_pred):
    predictions, labels = eval_pred
    if task != "stsb":
        predictions = np.argmax(predictions, axis=1)
    else:
        predictions = predictions[:, 0]
    return metric.compute(predictions=predictions, references=labels)

validation_key = "validation_mismatched" if task == "mnli-mm" else "validation_matched" if task == "mnli" else "validation" 
trainer = Trainer(
    model,
    args,
    train_dataset=encoded_dataset["train"],
    eval_dataset=encoded_dataset[validation_key],
    tokenizer=tokenizer,
    compute_metrics=compute_metrics
)

trainer.train()



TrainOutput(global_step=4210, training_loss=0.21714420998181413)
trainer.evaluate()

{'epoch': 1.0,
 'eval_accuracy': 0.9254587155963303,
 'eval_loss': 0.25063103437423706}
Now we just need to convert our dataset into the right format so that the model can work properly. We will use a small subset from Amazon review dataset in the fashion category. You can find the dataset here. The labels are still in the form of rating, so we need to change them into whether positive or negative. Reviews with 3 or more stars will be classified as positive, and the rest are negative. This is just for an example, feel free to change it the way you like.
After that, we split them into train, validation, and test and tokenize them using AutoTokenizer. We also need to convert our data to dataset object by subclassing torch.utils.data.Dataset object and implementing __len__ and __getitem__. Take a look at AmazonDataset class below. For training, just repeat the steps in the previous section. But this time, we use DistilBert instead of BERT. It is a small version of BERT. Faster and lighter!
As you can see, the evaluation is quite good (almost 100% accuracy!). Apparently, it’s because there are a lot of repetitive data. Some reviews can appear more than three times in the dataset. So, make sure that your data is clear and good enough to represent the actual world.
Even better, they also support hyperparameter search using Optuna or Ray tune (you can choose one). It will run the training process several times so it needs to have the model defined via a function (so it can be reinitialized at each new run). See model_init function below.
Besides that, it will also take a very long time to run. Alternatively, you can do a hyperparameter search using only a portion of the training data to save time and resources. After getting the best configuration, we can rerun the training using full data with the best configuration. Just do something like this:
train_dataset = encoded_dataset["train"].shard(index=1, num_shards=10)
This process will return a BestRun object containing information about the hyperparameter which is used for the best run. To use this configuration, just set the hyperparameter into TrainingArgument.
That’s it! If you want to try another task or another pretrained model or even use your own dataset, you can easily customize it to your needs by modifying a couple of lines, and BOOM! You already had your own transformers-powered NLP model!
[1] Huggingface Transformers
[2] Devlin et al., BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding (2018)
If you enjoyed reading this post and would like to hear more from me and other writers here, join Medium and subscribe to my newsletter. Or simply follow the links below. Thank you!