Full Guide to BERT For Q&A | Towards Data Science

Question-answering (Q&A) transformers are widely applicable, insanely cool applications of modern-NLP.
At first glance, most of us would view building something like this as a feat of great difficulty. Fortunately, most of us would be wrong.
Transformers are — despite their incredible performance — surprisingly straightforward to train or fine-tune to a specific task.
Couple that with the many massive language datasets available online, and we have all we need for building some incredible tools.
We’ll explain the key steps needed to train a Bert transformer model from scratch, and how we then fine-tune that model for Q&A. A key ingredient here are datasets — which we’ll also outline throughout the article.
First, what does the transformer training process look like? Although these models are incredibly powerful, the training process is surprisingly simple.
The first thing we need to understand is that there is a core Bert model. This core model acts as the central transformer engine — it is this engine that builds an understanding of language.
The output of this central block is not particularly useful by itself. It is simply a linguistic comprehension engine, it cannot tell you whether a sentence is happy or sad, identify entities, or answer our questions.
For all of these tasks — and more — we need a trained transformer head.
The head of a transformer is attached to the core engine. The core engine consumes some text, and processes it into meaningful numerical vectors. Those vectors are the information-rich input to our transformer heads.
The structure of each transformer head is built for a specific task, many consist of nothing more than a couple of feedforward neural network layers — organized in a way that we produce meaningful functionality once the head has been trained.
Therefore, to build a Q&A Bert model from scratch we must:
Train a core Bert engine.
Train a Q&A Bert Q&A head.
Sometimes there may be no existing Bert model that uses your language. If this is the case you must also train a Bert WordPiece tokenizer. I have written about this here.
For the sake of brevity, we’ll focus only on the two model training tasks here. Let’s jump into the core Bert training process.
Bert was initially trained using two concurrent processes. Masked-Language Modeling (MLM) and Next Sentence Prediction (NSP). Both need a lot of training data — which must be in the form of unstructured text.
Fortunately, if there’s one thing that we have plenty of on the internet — it’s unstructured text.
For MLM and NSP we need nothing more than a large dataset of unstructured text data. In terms of multi-lingual unstructured text, there is no better source than the OSCAR corpus.
There are (currently) 166 languages covered by OSCAR, ranging from English to the Nahuatl languages. A full list can be found here.
The easiest way to download data from OSCAR is via HuggingFace’s datasets library. Personally, I would 100% recommend taking this approach. We can very quickly download — for example — the English subset of OSCAR like so:
And there are a huge number of other datasets available, which we can find over on HuggingFace’s datasets viewer.
When downloading smaller datasets, we can use the code above without issue — but larger datasets (particularly from OSCAR) can be huge — the Italian subset of OSCAR weighs in at a cool 69GB of data, and the English, only 1.8TB.
In that case, we can stream the data by adding the streaming=True parameter to load_datasets.
Then either build the full input pipeline with streaming=True , or just iterate through smaller parts of of the dataset and store that in-memory (often streaming=True can cause code issues, in that case the latter option is best).
If you need help on the specifics of downloading data via HF datasets — I cover all in this video:
Once the data has been downloaded, we move onto training the core model with MLM (and NSP if preferred).
MLM consists of providing Bert with a sentence where several words (or tokens) have been hidden using a mask token. This mask token does nothing more than hide the original word.
ORIGINAL: "flying fish flew by the space station"MASKED: "[MASK] fish flew by the [MASK] station"
The original sentence is then passed along as the target output of the core Bert model. By doing this Bert is forced to read the other words in the sentence and attempt to understand to context to correctly predict the masked words.
Alongside MLM, it was found that an additional training processed called NSP improved the performance of Bert on downstream tasks (those tasks requiring a specific transformer head).
While Bert is attempting to guess the true words behind mask tokens, it is also tasked with identifying the two sentences provided as input belong together or not.
What this means is, given two sentences — A and B. Is sentence B a logical continuation of sentence A?
Logical continuation (IsNext)A: "Alia went to the shop to buy pasta"B: "The shop was closed"Not logical continuation (NotNext)A: "Alia went to the shop to buy pasta"B: "The armadillo was not very happy with the lion"
Whether both MLM and NSP is used for training the core Bert model — or just MLM, depends on your preference. Training just MLM is simpler, and typically achieves ~90% of the possible performance of MLM and NSP.
In the video below we cover the training process for MLM (if you’d rather use MLM and NSP, find that here).
We have a fully trained core Bert model, we can take that core and add several heads that allow us to perform different tasks with the model. However, these heads will initially be untrained — and so we must train them!
If you prefer video, we cover everything here too:
For Q&A the most popular dataset is the Stanford Question and Answering Dataset (SQuAD). Alongside the original English version, there are several other languages now available.
Language options (on HF datasets at the time of writing)Spanish: squad_esPortuguese: squad_v1_ptItalian: squad_itKorean: [squad_kor_v1, squad_kor_v2]Thai: [iapp_wiki_qa_squad, thaiqa_squad]English: [squad, squad_v2, squadshifts, squad_adversarial]
To download the English SQuAD data, we use:
Language options (at the time of writing)Spanish: squad_esPortuguese: squad_v1_ptItalian: squad_itKorean: [squad_kor_v1, squad_kor_v2]Thai: [iapp_wiki_qa_squad, thaiqa_squad]English: [squad, squad_v2, squadshifts, squad_adversarial]
For each sample, our data can be broken into three components:
Question — a string containing the question that we will ask Bert.
Context — a larger sequence (paragraphs) that contain the answer to our question.
Answer — a slice of the context that answers our question.
Given a question and context, our Q&A model must read both and return the token positions of the predicted answer within the context.
Before we begin tokenization, training, etc — we need to reformat our answers feature into the correct format for training. Presently it looks like:
{'text': ['the answer is here'], 'answer_start': [71]}
The 71 represents the character position in our context string where the answer begins. We can get the answer range by simply adding the length of text to answer_start — and we do this first:
The data format is now ready for tokenization.
We need to tokenize the SQuAD data so that it is readable by our Bert model. For the context and question features we can do using the standard tokenizer() function:
Which encodes both our context and question strings into single arrays of tokens. This will act as the input to our Q&A training, but we have no targets yet.
Our targets are the start and end positions of the answer, which we previously built using the character start and end positions within the context strings. However, we will be feeding tokens into Bert, so we need to provide the token start and end positions.
To do this, we need to convert the character start and end positions into token start and end positions — easily done with our add_token_positions function:
This function adds two more tensors to our Encoding object (which we feed into Bert) — the start_positions and end_positions.
Our tensors are now ready for training the Bert Q&A head.
We will be training using PyTorch, which means we will need to convert the tensors we’ve built into a PyTorch Dataset object.
We will feed our Dataset to our Q&A training loop using a Dataloader object, which we initialize with:
And finally, we setup our model parameters and begin the training loop.
After training our model all we need to do is save it!
And we’re done, we’ve trained a Q&A model from scratch.
That’s all for this article, we’ve covered the essentials behind training both the core and head of a Bert transformer model for Q&A and explored some of the best datasets we can use for both.
I hope you enjoyed it! If you have any questions, let me know via Twitter or in the comments below. If you’d like more content like this, I post on YouTube too.
Thanks for reading!
🤖 70% Discount on the NLP With Transformers Course
*All images are by the author except where stated otherwise