BERT NLP — How To Build a Question Answering Bot | by Michel Kana, Ph.D | Towards Data Science

This article will present key ideas about creating and coding a question answering system based on a neural network. The implementation uses Google’s language model known as pre-trained BERT. Hands-on proven PyTorch code for question answering with BERT fine-tuned and SQuAD is provided at the end of the article.
In Question Answering tasks, the model receives a question regarding text content and is required to mark the beginning and end of the answer in the text.
In the example above, the answer to the question “Where else besides the SCN cells are independent circadian rhythms also found?” is found at the position highlighted with red color. The question “What is the term for the independent clocks?” is answered at the blue position.
If we have a very large set of such texts together with sample questions and the position of the answers in the text, we can train a neural network to learn relationships between context, questions, and answers. The resulting network would be able to answer unseen questions given new contexts which are similar to the training texts.
Machine reading comprehension has captured the minds of computer scientists for decades. The recent production of large-scale labeled datasets has allowed researchers to build supervised neural systems that automatically answer questions posed in a natural language.
The Stanford Question Answering Dataset (SQuAD) is a prime example of large-scale labeled datasets for reading comprehension. Rajpurkar et al. developed SQuAD 2.0, which combines 100,000 answerable questions with 50,000 unanswerable questions about the same paragraph from a set of Wikipedia articles. The unanswerable questions were written adversarially by crowd workers to look similar to answerable ones.
doc_tokens describes the context, i.e. the text which we want our model to understand.
At the 52nd Annual Grammy Awards, Beyoncé received ten nominations, including Album of the Year for I Am... Sasha Fierce, Record of the Year for "Halo", and Song of the Year for "Single Ladies (Put a Ring on It)", among others. She tied with Lauryn Hill for most Grammy nominations in a single year by a female artist. In 2010, Beyoncé was featured on Lady Gaga's single "Telephone" and its music video. The song topped the US Pop Songs chart, becoming the sixth number-one for both Beyoncé and Gaga, tying them with Mariah Carey for most number-ones since the Nielsen Top 40 airplay chart launched in 1992. "Telephone" received a Grammy Award nomination for Best Pop Collaboration with Vocals.
question_text describes the question which should be answered from the context.
How many awards was Beyonce nominated for at the 52nd Grammy Awards?
orig_answer_text represents the correct answer to the question.
ten
The answer is always a portion from the context starting at start_position and ending at end_position. If the question does not have any answer in the context, is_impossible has the value true.
The SQuAD dataset offers 150,000 questions, which is not that much in the deep learning world. The idea behind transfer learning is to take a model that was trained on a very large dataset, then fine-tune that model using the SQuAD dataset.
BERT is a trained Transformer Encoder stack, with twelve in the Base version, and twenty-four in the Large version. BERT was trained on Wikipedia and Book Corpus, a dataset containing +10,000 books of different genres. I cover the Transformer architecture in detail in my article below.
towardsdatascience.com
We can use BERT to extract high-quality language features from the SQuAD text just by adding a single linear layer on top. The linear layer has two outputs, the first for predicting the probability that the current subtoken is the start of the answer and the second output for the end position of the answer.
Below you can find a summary of the model. I only display the first and last encoder layers for simplicity’s sake. The same layer would normally be repeated 12 times. We can see the BertEmbeddings layer at the beginning, followed by a Transformer architecture for each encoder layer: BertAttention, BertIntermediate, BertOutput. In the end, we have the BertPooler and Linear layer for two outputs.
Training the model is relatively straightforward. The bottom layers have already great English word representation, and we only really need to train the top layer, with a bit of tweaking going on in the lower levels to accommodate our task for question answering. You can customize the code I provided in my article below for this purpose.
towardsdatascience.com
You might expect a F1-score of around 74%. If we check the current SQuAD 1.0 leaderboard, we’ll see that this evaluation of the test dataset puts us in Top 100, which is acceptable given the limited resources available on free GPUs.
The fine-tuned model can be used to run inference on text and questions of our choice.
In this article, I explained how to fine-tune a pre-trained BERT model on the SQUaD dataset for solving question answering task on any text.
You can adapt my PyTorch code for NLU with BERT to solve your question-answering task.
Some readers might find the full code in this Google Colab Notebook more straight-forward. Credits go to Prachur Bhargava, Lead Data Scientist at Microsoft, and Himanshu Mohan, Software Engineer at American Express, who elegantly solved memory issues around training the model on Google Colab. Thank you, Prachur and Himanshu!
You can learn more about language models in my articles below.
towardsdatascience.com
towardsdatascience.com
towardsdatascience.com
towardsdatascience.com
towardsdatascience.com
towardsdatascience.com
towardsdatascience.com
Thanks for reading.