How-to Fine-Tune a Q&A Transformer | by James Briggs | Towards Data Science

Transformer models are undoubtedly the leaders in NLP — outperforming almost every other model architecture in almost every other task.
One of the most fascinating language-based tasks that transformers have proven themselves more than capable of dealing with is question-and-answering.
In this article, we will learn about the field of Q&A in NLP — and learn how to download and fine-tune some of the most advanced transformer models in the world for this very purpose. In short:
> Our Dataset  - Downloading SQuAD> Preparing The Data  - Extraction  - Encoding  - Initializing the Dataset> Fine-Tuning> Measuring Performance  - Implementing Accuracy
If you prefer video — I cover everything here too:
For our example, we will use the Stanford Question Answering Dataset (SQuAD) 2.0. Our model will be expected to correctly select the specific part of a text segment that answers a question we provide.
We can download the data in Python like so:
Which will return two files, our training data train-v2.0.json — and our validation data dev-v2.0.json. Both look like this:
Here we have three key parts in our data, which are:
Questions — strings containing the question which we will ask our model
Contexts — larger segments of text that contain the answers to our questions within them
Answers — shorter strings which are ‘extracts’ of the given contexts that provide an answer to our questions
So, we will provide the model with a question and corresponding answer. The model must then read both and return the token positions of the predicted answer from the context.
The first step in preparing this data for fine-tuning is extracting our questions, contexts, and answers from the JSON files into training and validation sets. We can do this like so:
This gives us our two datasets split between three lists (each):
Our contexts and questions are simple strings — each corresponds to each other. The answer to each question can be found within the context.
The answers lists are slightly different in that each item is a dictionary where the answer is contained within 'text' — and the starting position of this answer within the context is also provided in 'answer_start'.
This is okay, but we need to train our model to find the start and end of our answer within the context — so we need to add an 'answer_end' value too:
This gives us:
Our data is almost ready; we just need to convert our strings into tokens — and then translate our answer_start and answer_end indices from character-position to token-position.
Tokenization is easily done using a built-in HuggingFace tokenizer like so:
Our context-question pairs are now represented as Encoding objects. These objects merge each corresponding context and question strings to create the Q&A format expected by BERT — which is simply both context and question concatenated but separated with a [SEP] token:
This concatenated version is stored within the input_ids attribute of our Encoding object. But, rather than the human-readable text — the data is stored as BERT-readable token IDs.
The tokenizer is great, but it doesn’t produce our answer start-end token positions. For that, we define a custom add_token_positions function:
This function adds two more attributes to our Encoding objects — start_positions and end_positions. Each of these is simply a list containing the start/end token positions of the answer that corresponds to their respective question-context pairs.
We’ve now prepared our data, and we have everything we need — we just need to transform it into the correct format for training with PyTorch.
For this, we need to build a dataset object, which we can do easily like so:
We will be able to feed these dataset objects into our model during training and validation.
Our data is now wholly ready for use by our model. All we do now is set up our PyTorch environment, initialize the DataLoader which we will be using to load data during training — and finally, begin fine-tuning our model:
DistilBert is a small model compared to most transformer models — but this will still take some time to train.
Once done, we can go ahead and save our newly fine-tuned model (and tokenizer if needed) — we will save it to a new directory models/distilbert-custom:
When loading our model again, we can use the same from_pretrained method that we use when loading models from HuggingFace — all we do is replace the online model path/name with our local path — in this case:
model_path = 'models/distilbert-custom'model = DistilBertForQuestionAnswering.from_pretrained(model_path)tokenizer = DistilBertTokenizerFast.from_pretrained(model_path)
Or using the generic AutoModel loader:
model = AutoModel.from_pretrained(model_path)tokenizer = AutoTokenizer.from_pretrained(model_path)
Once we have trained our model, we can begin making predictions and asking our model a few questions.
Bear in mind here that we are already using a highly optimized base model — which gets us halfway there — but we could still benefit from experimenting with different models.
But for now, we just need to learn this approach — and how to apply it to our own more specific use-cases. So let’s learn what to do post-training.
To extract the start-end token range from our model, we can access the start_logits and end_logits tensors and perform an argmax function like so:
start_pred = torch.argmax(outputs['start_logits'], dim=1)end_pred = torch.argmax(outputs['end_logits'], dim=1)
The simplest metric we can measure here is accuracy — more specifically known as exact match (EM) — did the model get the right start or end token, or not? Of course, this doesn’t factor in models that are getting close and maybe missing by one or two tokens — but it’s a start.
To calculate the EM of each batch, we take the sum of the number of matches per batch — and divide by the total. We do this with PyTorch like so:
acc = ( (start_pred == start_true).sum() / len(start_pred) ).item()
The final .item() extracts the tensor value as a plain and simple Python int.
Now we know how to calculate our Q&A accuracy — let’s implement it for our val_dataset!
Once again, we set up a DataLoader — and then loop through each batch. This time though, we call model.eval() to switch the behavior of relevant layers from training to inference mode — and use torch.no_grad() to stop PyTorch from calculating model gradients (only needed during training):
After all of that, we have an — albeit rough — measure of accuracy.
63.6% of the time, the model manages to get the exact span (start and/or end) of our answer correct. 63.6% is certainly not a particularly impressive number until we look a little deeper into how the model performs.
The EM metric doesn’t paint a full picture — despite scoring an EM of just 18.75% in the last batch — the model did get incredibly close in 4/8 of the questions, which is not reflected in the EM score.
Nonetheless, when measuring and comparing models — EM is the perfect place to start.
That’s all for this walk-through! We’ve taken a pre-trained DistilBert model, fitted it with a Q&A head — and fine-tuned it using the SQuAD dataset. Producing our very own Q&A model.
However, more importantly, we have introduced the process of taking a Q&A dataset — and used it to ‘fine-tune’ a pre-trained transformer model.
Learning and understanding this process is very important and, when applied correctly, can push Q&A model performance on specific use-cases much higher.
You most likely won’t be using the SQuAD dataset for any fine-tuning — instead, you will be using your own data, or an open-source Q&A dataset:
NewsQA — 120K CNN news articles
FigureQA — Chart-based Q&A
FiQA — Financial opinion Q&A from Reddit
CommonSenseQA — Good luck with this one 😄
There are already a tremendous number of datasets available — and more are made available frequently — so there’s plenty to learn from!
I hope you enjoyed this article! If you have any questions, let me know via Twitter or in the comments below. If you’d like more content like this, I post on YouTube too.
Thanks for reading!
Fine-tuning with custom datasets, HuggingFace’s Transformers Docs
🤖 NLP With Transformers Course
Full Article Code
*All images are by the author except where stated otherwise