Generate boolean (yes/no) questions from any content using T5 text-to-text transformer model | by Ramsri Goutham | Towards Data Science

The input to our program will be any general content/paragraph -
Months earlier, Coca-Cola had begun “Project Kansas.” It sounds like a nuclear experiment but it was just a testing project for the new flavor. In individual surveys, they’d found that more than 75% of respondents loved the taste, 15% were indifferent, and 10% had a strong aversion to the taste to the point that they were angry.
The output will be boolean (yes/no) questions generated from the above input.
Boolean (yes/no) questions generated from the T5 Model :
1: Does coca cola have a kansas flavor?2: Is project kansas a new coca cola flavor?3: Is project kansas the same as coca cola?
Today we will see how we can train a T5 model from Huggingface’s transformers library to generate these boolean questions. We will also see how we can use the pre-trained model provided to generate these boolean (yes/no) questions.
Imagine a scenario where, as a student, you interact with a chatbot to learn a concept interactively. The chatbot presents you with relevant byte-sized snippets of content from a textbook chapter based on your responses. It also wants to assess in real-time whether you have understood the topic presented. It is impractical to manually pre-generate assessments for each snippet of the textbook chapter. The chatbot can make use of this algorithm to generate boolean(yes/no) questions in real-time and evaluate your understanding of the topic.
Let’s get started —
I used the BoolQ dataset to collect the passage, question and answer triplets, and prepared training and validation sets.
The boolQ dataset has the following format -
{  "question": "is france the same timezone as the uk",  "passage": "At the Liberation of France in the summer of 1944, Metropolitan France kept GMT+2 as it was the time then used by the Allies (British Double Summer Time). In the winter of 1944--1945, Metropolitan France switched to GMT+1, same as in the United Kingdom, and switched again to GMT+2 in April 1945 like its British ally. In September 1945, Metropolitan France returned to GMT+1 (pre-war summer time), which the British had already done in July 1945. Metropolitan France was officially scheduled to return to GMT+0 on November 18, 1945 (the British returned to GMT+0 in on October 7, 1945), but the French government canceled the decision on November 5, 1945, and GMT+1 has since then remained the official time of Metropolitan France."  "answer": false,  "title": "Time in France",}
There is a “passage” which has a corresponding “question” and the correct boolean “answer” — true or false.
We will discuss in detail how you can -
Use my pre-trained model to generate boolean questions for any given content.Use my training code and dataset to replicate the results on your own GPU machine.
Use my pre-trained model to generate boolean questions for any given content.
Use my training code and dataset to replicate the results on your own GPU machine.
T5 is a new transformer model from Google that is trained in an end-to-end manner with text as input and modified text as output. You can read more about it here.
It achieves state-of-the-art results on multiple NLP tasks like summarization, question answering, machine translation, etc using a text-to-text transformer trained on a large text corpus.
I gave the “passage” and “answer” as input to my T5 transformer model and trained it to generate the “question” as output.
All the code for using pre-trained model and training the model with given data is available at -
github.com
The Python file t5_inference.py contains all the code presented below.
First, install the necessary libraries -
!pip install torch==1.4.0!pip install transformers==2.9.0!pip install pytorch_lightning==0.7.5
Run inference with any text/paragraph as input and see the boolean questions generated -
The output from the above code is -
Context:  Months earlier, Coca-Cola had begun “Project Kansas.” It sounds like a nuclear experiment but it was just a testing project for the new flavor. In individual surveys, they’d found that more than 75% of respondents loved the taste, 15% were indifferent, and 10% had a strong aversion to the taste to the point that they were angry.Beam decoding [Most accurate questions] ::Does coca cola have a kansas flavor?Is project kansas the same as coca cola?Is project kansas a new coca cola flavor?TopKP decoding [Not very accurate but more variety in questions] ::Does coca cola have a koala flavor?Is kakao the same as project kansas?Was project ksoda a real thing?Time elapsed  1.2351574897766113
All the training code and dataset used for training are available in the Github repo mentioned. We will go through the steps that I used to train the model.
The file boolQ_prepare_train_validation_dataset.ipynb contains all the code to prepare the train and validation datasets. I took the boolQ dataset which is available as a JSON file and converted it into a csv file.
Thanks to Suraj Patil for the amazing Colab notebook on training T5 for any text-to-text task. I borrowed most of the training code from the Colab notebook, changing only the dataset class and training parameters. I adapted the dataset class to our boolQ dataset.
The training code is available as train.py in the Github Repo.
All you need to do is clone the repo on any GPU machine, install requirements.txt, and run train.py to train the T5 model.
Training this model for 4 epochs (default) took about 5–6 hrs on p2.xlarge (AWS ec2).
The dataset class looks like below —
The key is how we give our input and output to the T5 model trainer. I gave the “passage” and “answer” as input to my T5 transformer model and trained it to generate the “question” as output as shown below -
Input format to T5 for training
truefalse: yes passage: At the Liberation of France in the summer of 1944, Metropolitan France kept GMT+2 as it was the time then used by the Allies (British Double Summer Time). In the winter of 1944--1945, Metropolitan France switched to GMT+1, same as in the United Kingdom, and switched again to GMT+2 in April 1945 like its British ally. In September 1945, Metropolitan France returned to GMT+1 (pre-war summer time), which the British had already done in July 1945. Metropolitan France was officially scheduled to return to GMT+0 on November 18, 1945 (the British returned to GMT+0 in on October 7, 1945), but the French government canceled the decision on November 5, 1945, and GMT+1 has since then remained the official time of Metropolitan France. 

Output format to T5 for training
Is france the same timezone as the uk? 

Note: The text “truefalse: yes” or “truefalse: no” is supposed to generate an appropriate boolean question to which the answer is “yes” or “no” as given in the text. But with my trails to train T5, it hasn’t been effective. So make sure that you don’t rely on the initial token “yes” or “no” given as an answer to the boolean question generated.
The total dataset is only about ~10k samples hence the quality of questions generated is sometimes not very great. Please note that.
Happy coding!
I am running a 4-week cohort-based course on “Practical Introduction to NLP” with Maven, the world’s best platform for cohort-based learning. If you wish to transform yourself from a Python developer to a junior NLP developer with practical project experience in 4 weeks, grab your spot now!
I launched a very interesting Udemy course titled “Question generation using NLP” expanding on some of the techniques discussed in this blog post. If you would like to take a look at it, here is the link.
Feel free to connect on Linkedin if you have any questions or you just wanted to say hi!