Practical AI : Automatically Generate Multiple Choice Questions (MCQs) from any content with BERT Summarizer, Wordnet and Conceptnet | by Ramsri Goutham | Towards Data Science

If you want to try a live demo of MCQ generation in action, please visit https://questgen.ai/
In this post we will see how to automatically generate Multiple Choice Questions (MCQs) from any story or article. This is one step towards automatically generating those MCQs you see in Middle school/ High school.
Input: The input to our program will be an article (example article provided in the github repository below) like the following —
The Greek historian knew what he was talking about. The Nile River fed Egyptian civilization for hundreds of years. The Longest River the Nile is 4,160 miles long — the world’s longest river. It begins near the equator in Africa and flows north to the Mediterranean Sea ..............
Output: The output will be a set of automatically generated MCQs with efficient distractors (wrong answer choices) -
1) The Nile provided so well for  _______  that sometimes they had surpluses, or more goods than they needed.	 a )   Angolan	 b )   Algerian	 c )   Egyptians	 d )   BantuMore options:  ['Basotho', 'Beninese', 'Berber', 'Black African', 'Burundian', 'Cameroonian', 'Carthaginian', 'Chadian', 'Chewa', 'Congolese', 'Djiboutian', 'Egyptian', 'Ethiopian', 'Eurafrican', 'Ewe', 'Fulani'] 2) As in many ancient societies, much of the knowledge of  _______  came about as priests studied the world to find ways to please the gods.	 a )   Malawi	 b )   East Africa	 c )   Somalia	 d )   EgyptMore options:  ['Togo', 'Zimbabwe', 'Gabon', 'Ghana', 'Lake Tanganyika', 'Ottoman Empire', 'Mozambique', 'Iran', 'Israel', 'Saudi Arabia', 'Lebanon', 'Turkey', 'Iraq', 'Levant', 'Syria', 'Jordan'] 3) The  _______  provided so well for Egyptians that sometimes they had surpluses, or more goods than they needed.	 a )   Nyala	 b )   Omdurman	 c )   Nile	 d )   Port SudanMore options:  ['Khartoum', 'Nubian Desert', 'Darfur', 'Libyan Desert', 'Kordofan', 'Gulu', 'Buganda', 'Entebbe', 'Jinja', 'Lake Edward', 'entebbe', 'gulu', 'kayunga', 'Upper Egypt', 'Suez Canal', 'Aswan High Dam']
Let’s get started to see how you can build your own MCQ generator leveraging state of the art NLP.
If you want to try out advanced MCQ generator live, please visit https://questgen.ai/
All the code and the jupyter notebook is available at -
github.com
First install the necessary libraries in the jupyter notebook. If you encounter any error just play around with versions and check other incompatibilities :
!pip install gensim!pip install git+https://github.com/boudinfl/pke.git!python -m spacy download en!pip install bert-extractive-summarizer --upgrade --force-reinstall!pip install spacy==2.1.3 --upgrade --force-reinstall!pip install -U nltk!pip install -U pywsdimport nltknltk.download('stopwords')nltk.download('popular')
Here we use a simple library bert-extractive-summarizer that does the job for us. We load the total text from the egypt.txt file present in the Github repo and ask the library to give us summarized text. You can play with the parameters of the ratio, max and min length of sentences to be kept for summarization etc
from summarizer import Summarizerf = open("egypt.txt","r")full_text = f.read()model = Summarizer()result = model(full_text, min_length=60, max_length = 500 , ratio = 0.4)summarized_text = ''.join(result)print (summarized_text)
We get a summarized portion of the original text as output :
The Nile River fed Egyptian civilization for hundreds of years. It begins near the equator in Africa and flows north to the Mediterranean Sea. A delta is an area near a river’s mouth where the water deposits fine soil called silt.......................
We use python keyword extractor (PKE) library and extract all the important keywords from the original text. Then keep only those keywords that are present in the summarized text. Here I am only extracting nouns as they will be more suitable for MCQs. Also I am only extracting 20 keywords but you can play around with that parameter (get_n_best).
The output is -
Original text keywords: ['egyptians', 'nile river', 'egypt', 'nile', 'euphrates', 'tigris', 'old kingdom', 'red land', 'crown', 'upper', 'lower egypt', 'narmer', 'longest river', 'africa', 'mediterranean sea', 'hyksos', 'new kingdom', 'black land', 'ethiopia', 'middle kingdom']Keywords present in summarized text: ['egyptians', 'nile river', 'egypt', 'nile', 'old kingdom', 'red land', 'crown', 'upper', 'lower egypt', 'narmer', 'africa', 'mediterranean sea', 'new kingdom', 'middle kingdom']
For each of the keyword we will extract corresponding sentences that has the word from the summarized text.
The sample output is —
{'egyptians': ['The Nile provided so well for Egyptians that sometimes they had surpluses, or more goods than they needed.', 'For example, some ancient Egyptians learned to be scribes, people whose job was to write and keep records.', 'Egyptians believed that if a tomb was robbed, the person buried there could not have a happy afterlife.', 'nile river': ['The Nile River fed Egyptian civilization for hundreds of years.'],  'old kingdom': ['Historians divide ancient Egyptian dynasties into the Old Kingdom, the Middle Kingdom, and the New Kingdom.', 'The Old Kingdom started about 2575 B.C., when the Egyptian empire was gaining strength.'], }        .....................
Here we get distractors (wrong answer choices) from Wordnet and Conceptnet to generate our final MCQ Questions.
What are distractors (wrong answer choices) ?
If let’s say “Historians divide ancient Egyptian dynasties into the Old Kingdom, the Middle Kingdom, and the New Kingdom.” is our sentence and we want to give “Egyptian” as the fill in the blank key word then the other wrong answer choices should be similar to Egyptian but not synonymous for Egyptian.
Eg: Historians divide ancient __________  dynasties into the Old Kingdom, the Middle Kingdom, and the New Kingdoma)Egyptianb)Ethiopianc)Angoliand)Algerian
In the above question options b) , c) and d) are distractors.
How do we generate these distractors automatically ?
We take two approaches one with wordnet and the other with conceptnet to get distractors.
Wordnet Approach used in our code :
Given input as a sentence and keyword, we get the “sense” of the word first.
Let me explain with “sense” with an example. If we have a sentence “ The bat flew into the jungle and landed on a tree” and a keyword “bat”, we automatically know that here we are talking about the mammal bat that has wings not cricket bat or baseball bat. Although we humans are good at it, the algorithms are not very good at distinguishing one vs the other. This is called word sense disambiguation (WSD). In wordnet “bat” may have several senses one for cricket bat, one for flying mammal etc. So the function get_wordsense tries to get the correct sense of the word given a sentence along with it. This is not perfect all the times hence we get some errors if the algorithm narrows onto a wrong sense. Then distractors (wrong answer choices) will also be wrong.
Once we identify the sense we call get_distractors_wordnet function to get distractors. What happens here is that let’s say we get a word like “cheetah” and identify its sense, we then go to its hypernym. A hypernym is a higher level category for a given word. In our example feline is hypernym for cheetah.
Then we go to find all hyponyms ( sub categories) of feline which might be Leopard, Tiger, Lion all belonging to the feline group. So we can use Leopard, Tiger, Lion as distractors (wrong answer choices) for the given MCQ.
Conceptnet Approach used in our code :
Not all words are available in wordnet and not all have hypernyms. Hence if we fail with wordnet we look in conceptnet as well for distractors. get_distractors_conceptnet is the main function used to get distractors from conceptnet. Conceptnet doesn’t have provision to disambiguate between different word senses as we discussed above with the example of bat. Hence we need to go ahead with whatever sense conceptnet gives us when we query with a given word.
Let’s see how we use conceptnet in our use case. We don’t install anything because we use the conceptnet API directly. Note that there is an hourly API rate limit so beware of it.
Given a word like “California” we query conceptnet with it and retrieve the “Partof” relationship. In our example “california” is partof “United States”.
Now we go to “United States” and see what other things does it share a “partof” relationship with. That would be other states like “Texas”, “Arizona”, “Seattle’ etc. Hence for our query word “California” we fetched the distractors “Texas”, “Arizona” etc
Finally we generate output as follows :
#############################################################################NOTE::::::::  Since the algorithm might have errors along the way, wrong answer choices generated might not be correct for some questions. #############################################################################1) The Nile provided so well for  _______  that sometimes they had surpluses, or more goods than they needed.	 a )   Angolan	 b )   Algerian	 c )   Egyptians	 d )   BantuMore options:  ['Basotho', 'Beninese', 'Berber', 'Black African', 'Burundian', 'Cameroonian', 'Carthaginian', 'Chadian', 'Chewa', 'Congolese', 'Djiboutian', 'Egyptian', 'Ethiopian', 'Eurafrican', 'Ewe', 'Fulani'] 2) As in many ancient societies, much of the knowledge of  _______  came about as priests studied the world to find ways to please the gods.	 a )   Malawi	 b )   East Africa	 c )   Somalia	 d )   EgyptMore options:  ['Togo', 'Zimbabwe', 'Gabon', 'Ghana', 'Lake Tanganyika', 'Ottoman Empire', 'Mozambique', 'Iran', 'Israel', 'Saudi Arabia', 'Lebanon', 'Turkey', 'Iraq', 'Levant', 'Syria', 'Jordan'] 3) The  _______  provided so well for Egyptians that sometimes they had surpluses, or more goods than they needed.	 a )   Nyala	 b )   Omdurman	 c )   Nile	 d )   Port SudanMore options:  ['Khartoum', 'Nubian Desert', 'Darfur', 'Libyan Desert', 'Kordofan', 'Gulu', 'Buganda', 'Entebbe', 'Jinja', 'Lake Edward', 'entebbe', 'gulu', 'kayunga', 'Upper Egypt', 'Suez Canal', 'Aswan High Dam'] 4) It combined the red  _______  of Lower Egypt with the white  _______  of Upper Egypt.	 a )   Capital	 b )   Crown	 c )   Masthead	 d )   HeadMore options:  [] 5) It combined the red Crown of Lower Egypt with the white Crown of  _______  Egypt.	 a )   Upper Berth	 b )   Lower Berth	 c )   UpperMore options:  []
Note that there could be many errors either due to wrong sense of the word that is extracted from Wordnet word sense disambiguation (WSD) algorithm or conceptnet identifying it with a different sense due to its limitation.
Just like “partof” relationship, there is “IsA” relationship in Conceptnet that could be further explored to get distractors.
Just like “partof” relationship, there is “IsA” relationship in Conceptnet that could be further explored to get distractors.
You could use these for IsA relationship -
QueryUrl = “http://api.conceptnet.io/query?node=%s&rel=/r/IsA&end=%s&limit=10"%(link,link)
Retrieval URL = “http://api.conceptnet.io/query?node=/c/en/%s/n&rel=/r/IsA&start=/c/en/%s&limit=5"%(word,word)
2. Since we have multiple sentences for each keyword use all of them to do word sense disambiguation (WSD) and choose the sense with the highest count.
3. Can alternatively use word vectors (word2vec, glove) as a way of distractors for a given word.
4. You can use pronoun resolution (neural coreference resolution) on the full text before passing it to BERT summarizer. Then any sentences with pronouns shall be resolved so that when presented as MCQ it looks complete and independent.
I am running a 4-week cohort-based course on “Practical Introduction to NLP” with Maven, the world’s best platform for cohort-based learning. If you wish to transform yourself from a Python developer to a junior NLP developer with practical project experience in 4 weeks, grab your spot now!
I launched a very interesting Udemy course titled “Question generation using NLP” expanding on some of the techniques discussed in this blog post. If you would like to take a look at it, here is the link.